[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:29:42 dist-test-slave-mrck krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:29:42 dist-test-slave-mrck krb5kdc[399](info): set up 2 sockets
Sep 26 11:29:42 dist-test-slave-mrck krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 11:29:43.087422 392 mini_kdc.cc:121] Time spent starting KDC: real 2.761s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260926 11:29:43.296784 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.209s user 0.000s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260926 11:29:43.409554 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.113s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260926 11:29:43.661330 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.252s user 0.001s sys 0.000s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/krb5kdc/test-user.keytab.
W20260926 11:29:44.375867 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.714s user 0.000s sys 0.002s
Sep 26 11:29:44 dist-test-slave-mrck krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:29:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:29:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:29:44.420364 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35449
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35449
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:29:44.525421 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:29:44.525727 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:29:44.525799 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:29:44.529760 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:29:44.529847 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:29:44.529886 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:29:44.529959 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:29:44.529994 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:29:44.535154 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35449
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:29:44.536202 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:29:44.537207 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:29:44.546170 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:29:44.546193 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:29:44.546219 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:29:44.546175 412 server_base.cc:1034] running on GCE node
I20260926 11:29:44.547041 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:29:44.548244 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:29:44.549423 412 hybrid_clock.cc:648] HybridClock initialized: now 1790422184549358 us; error 84 us; skew 500 ppm
Sep 26 11:29:44 dist-test-slave-mrck krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:29:44.555699 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:29:44.556668 412 webserver.cc:460] Webserver started at http://127.0.98.62:33259/ using document root <none> and password file <none>
I20260926 11:29:44.557230 412 fs_manager.cc:359] Metadata directory not provided
I20260926 11:29:44.557317 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:29:44.557546 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:29:44.559537 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "d414c5d717dc48129d400307e129f84e"
format_stamp: "Formatted at 2026-09-26 11:29:44 on dist-test-slave-mrck"
server_key: "7ade6c39aad2f01c9a77a3abc9bb05d8"
server_key_iv: "d8a5e9324337ee1bbfae8b73a8950511"
server_key_version: "encryptionkey@0"
I20260926 11:29:44.560137 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "d414c5d717dc48129d400307e129f84e"
format_stamp: "Formatted at 2026-09-26 11:29:44 on dist-test-slave-mrck"
server_key: "7ade6c39aad2f01c9a77a3abc9bb05d8"
server_key_iv: "d8a5e9324337ee1bbfae8b73a8950511"
server_key_version: "encryptionkey@0"
I20260926 11:29:44.563928 412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 11:29:44.566265 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:44.567494 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:29:44.567642 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "d414c5d717dc48129d400307e129f84e"
format_stamp: "Formatted at 2026-09-26 11:29:44 on dist-test-slave-mrck"
server_key: "7ade6c39aad2f01c9a77a3abc9bb05d8"
server_key_iv: "d8a5e9324337ee1bbfae8b73a8950511"
server_key_version: "encryptionkey@0"
I20260926 11:29:44.567775 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:29:44.596294 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:29:44.599211 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:29:44.599427 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:29:44.606349 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35449
I20260926 11:29:44.606376 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35449 every 8 connection(s)
I20260926 11:29:44.607400 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:29:44.610124 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:29:44.615489 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d414c5d717dc48129d400307e129f84e: Bootstrap starting.
I20260926 11:29:44.617110 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 412
I20260926 11:29:44.617255 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:29:44.618069 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d414c5d717dc48129d400307e129f84e: Neither blocks nor log segments found. Creating new log.
I20260926 11:29:44.618542 392 external_mini_cluster.cc:1428] Setting key 50f4461380f8da36b05d8981e3912ff2
I20260926 11:29:44.618844 480 log.cc:824] T 00000000000000000000000000000000 P d414c5d717dc48129d400307e129f84e: Log is configured to *not* fsync() on all Append() calls
I20260926 11:29:44.620846 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d414c5d717dc48129d400307e129f84e: No bootstrap required, opened a new log
I20260926 11:29:44.625885 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P d414c5d717dc48129d400307e129f84e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d414c5d717dc48129d400307e129f84e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35449 } }
I20260926 11:29:44.626171 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P d414c5d717dc48129d400307e129f84e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:29:44.626228 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P d414c5d717dc48129d400307e129f84e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d414c5d717dc48129d400307e129f84e, State: Initialized, Role: FOLLOWER
I20260926 11:29:44.626701 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P d414c5d717dc48129d400307e129f84e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d414c5d717dc48129d400307e129f84e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35449 } }
I20260926 11:29:44.626873 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P d414c5d717dc48129d400307e129f84e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:29:44.626995 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P d414c5d717dc48129d400307e129f84e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:29:44.627115 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d414c5d717dc48129d400307e129f84e [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:29:44.628206 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P d414c5d717dc48129d400307e129f84e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d414c5d717dc48129d400307e129f84e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35449 } }
I20260926 11:29:44.628559 480 leader_election.cc:302] T 00000000000000000000000000000000 P d414c5d717dc48129d400307e129f84e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d414c5d717dc48129d400307e129f84e; no voters:
I20260926 11:29:44.628893 480 leader_election.cc:288] T 00000000000000000000000000000000 P d414c5d717dc48129d400307e129f84e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:29:44.628981 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d414c5d717dc48129d400307e129f84e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:29:44.629217 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P d414c5d717dc48129d400307e129f84e [term 1 LEADER]: Becoming Leader. State: Replica: d414c5d717dc48129d400307e129f84e, State: Running, Role: LEADER
I20260926 11:29:44.629536 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P d414c5d717dc48129d400307e129f84e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d414c5d717dc48129d400307e129f84e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35449 } }
I20260926 11:29:44.630223 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P d414c5d717dc48129d400307e129f84e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:29:44.630621 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P d414c5d717dc48129d400307e129f84e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d414c5d717dc48129d400307e129f84e. Latest consensus state: current_term: 1 leader_uuid: "d414c5d717dc48129d400307e129f84e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d414c5d717dc48129d400307e129f84e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35449 } } }
I20260926 11:29:44.630825 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P d414c5d717dc48129d400307e129f84e [sys.catalog]: This master's current role is: LEADER
I20260926 11:29:44.631042 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P d414c5d717dc48129d400307e129f84e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d414c5d717dc48129d400307e129f84e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d414c5d717dc48129d400307e129f84e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35449 } } }
I20260926 11:29:44.631147 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P d414c5d717dc48129d400307e129f84e [sys.catalog]: This master's current role is: LEADER
I20260926 11:29:44.631358 490 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:29:44.634399 490 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 11:29:44 dist-test-slave-mrck krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:29:44.639500 490 catalog_manager.cc:1347] Generated new cluster ID: 93e7c2965ad24fd48c248ac4d7bc10a4
I20260926 11:29:44.639560 490 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:29:44.645756 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:44.621620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42950 (local address 127.0.98.62:35449)
0926 11:29:44.622083 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:44.622103 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:44.623318 (+ 1215us) server_negotiation.cc:407] Connection header received
0926 11:29:44.627054 (+ 3736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:44.627068 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:44.627416 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:44.627709 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:44.630631 (+ 2922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:44.632284 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:44.634063 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:44.634753 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:44.634869 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:44.639890 (+ 5021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:44.639896 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:44.639910 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:44.639939 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:44.642081 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:44.642651 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:44.642657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:44.642662 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:44.642805 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:44.643193 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:44.643196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:44.643198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:44.643677 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:44.643945 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:44.644542 (+ 597us) server_negotiation.cc:299] Negotiation successful
0926 11:29:44.644823 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":301,"thread_start_us":136,"threads_started":1}
I20260926 11:29:44.659198 490 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:29:44.660152 490 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:29:44.663610 490 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d414c5d717dc48129d400307e129f84e: Generated new TSK 0
I20260926 11:29:44.664270 490 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:29:44.692939 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35449
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:29:44.812387 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:29:44.812667 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:29:44.812745 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:29:44.816753 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:29:44.816841 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:29:44.816931 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:29:44.821946 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35449
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:29:44.822978 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:29:44.824075 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:29:44.833889 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:29:44.833884 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:29:44.833878 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:29:44.834213 506 server_base.cc:1034] running on GCE node
I20260926 11:29:44.834643 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:29:44.835282 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:29:44.836431 506 hybrid_clock.cc:648] HybridClock initialized: now 1790422184836414 us; error 29 us; skew 500 ppm
Sep 26 11:29:44 dist-test-slave-mrck krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:29:44.839133 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:29:44.840003 506 webserver.cc:460] Webserver started at http://127.0.98.1:33937/ using document root <none> and password file <none>
I20260926 11:29:44.840507 506 fs_manager.cc:359] Metadata directory not provided
I20260926 11:29:44.840564 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:29:44.840720 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:29:44.842643 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-0/data/instance:
uuid: "f3cb5130c2414326af715226e9513cd8"
format_stamp: "Formatted at 2026-09-26 11:29:44 on dist-test-slave-mrck"
server_key: "8508be40325f19d63baec25861b59081"
server_key_iv: "59da69afb3e6149259183d67377ee2d1"
server_key_version: "encryptionkey@0"
I20260926 11:29:44.843149 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f3cb5130c2414326af715226e9513cd8"
format_stamp: "Formatted at 2026-09-26 11:29:44 on dist-test-slave-mrck"
server_key: "8508be40325f19d63baec25861b59081"
server_key_iv: "59da69afb3e6149259183d67377ee2d1"
server_key_version: "encryptionkey@0"
I20260926 11:29:44.846998 506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 11:29:44.849372 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:44.850593 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 11:29:44.850734 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "f3cb5130c2414326af715226e9513cd8"
format_stamp: "Formatted at 2026-09-26 11:29:44 on dist-test-slave-mrck"
server_key: "8508be40325f19d63baec25861b59081"
server_key_iv: "59da69afb3e6149259183d67377ee2d1"
server_key_version: "encryptionkey@0"
I20260926 11:29:44.850836 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:29:44.866725 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:29:44.869658 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:29:44.869850 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:29:44.870422 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:29:44.871289 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:29:44.871335 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:44.871373 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:29:44.871426 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:44.880415 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35001
I20260926 11:29:44.880460 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35001 every 8 connection(s)
I20260926 11:29:44.881273 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:29:44.889200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 506
I20260926 11:29:44.889319 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:29:44.889580 392 external_mini_cluster.cc:1428] Setting key af22946a187533fc1184e8724b9fbaab
Sep 26 11:29:44 dist-test-slave-mrck krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:29:44.895146 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:44.882975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53369 (local address 127.0.98.62:35449)
0926 11:29:44.883136 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:44.883140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:44.884019 (+ 879us) server_negotiation.cc:407] Connection header received
0926 11:29:44.885292 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:44.885294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:44.885341 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:44.885418 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:44.886559 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:44.887094 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:44.888297 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:44.888537 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:44.888622 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:44.891287 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:44.891290 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:44.891291 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:44.891313 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:44.892917 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:44.893550 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:44.893553 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:44.893554 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:44.893629 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:44.894032 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:44.894034 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:44.894035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:44.894197 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:44.894269 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:44.894919 (+ 650us) server_negotiation.cc:299] Negotiation successful
0926 11:29:44.895026 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 11:29:44.896487 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:44.883386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35449 (local address 127.0.98.1:53369)
0926 11:29:44.883862 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:44.883902 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:44.885103 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:44.885578 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:44.885585 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:44.885930 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:29:44.886407 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:44.886420 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:44.887208 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:44.887211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:44.888188 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:44.888197 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:44.888665 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:44.888667 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:44.888798 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:44.889491 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:29:44.889507 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:29:44.891112 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:29:44.893052 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:44.893058 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:44.893074 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:44.893441 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:44.893718 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:44.893724 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:44.893728 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:44.893919 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:44.894298 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:29:44.894307 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:29:44.894641 (+ 334us) client_negotiation.cc:769] Sending connection context
0926 11:29:44.894885 (+ 244us) client_negotiation.cc:240] Negotiation successful
0926 11:29:44.895276 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":101,"threads_started":1}
I20260926 11:29:44.897761 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35449
I20260926 11:29:44.898005 635 heartbeater.cc:464] Registering TS with master...
I20260926 11:29:44.898480 635 heartbeater.cc:511] Master 127.0.98.62:35449 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 11:29:44.900053 444 ts_manager.cc:194] Registered new tserver with Master: f3cb5130c2414326af715226e9513cd8 (127.0.98.1:35001)
I20260926 11:29:44.901234 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53369
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:29:44.916633 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35449
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:29:45.026739 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:29:45.027030 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:29:45.027103 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:29:45.031189 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:29:45.031263 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:29:45.031396 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:29:45.036654 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35449
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:29:45.037672 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:29:45.038648 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:29:45.048386 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:29:45.048409 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:29:45.048393 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:29:45.048571 640 server_base.cc:1034] running on GCE node
I20260926 11:29:45.049034 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:29:45.049634 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:29:45.050827 640 hybrid_clock.cc:648] HybridClock initialized: now 1790422185050797 us; error 51 us; skew 500 ppm
Sep 26 11:29:45 dist-test-slave-mrck krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:29:45.053619 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:29:45.054389 640 webserver.cc:460] Webserver started at http://127.0.98.2:37789/ using document root <none> and password file <none>
I20260926 11:29:45.055022 640 fs_manager.cc:359] Metadata directory not provided
I20260926 11:29:45.055112 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:29:45.055333 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:29:45.057348 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-1/data/instance:
uuid: "d10bca4433d845d992b9ae98c0841faf"
format_stamp: "Formatted at 2026-09-26 11:29:45 on dist-test-slave-mrck"
server_key: "b08ac860883f67de910878bfb255bbee"
server_key_iv: "d34404d3573db53161d21f4c11b3b09e"
server_key_version: "encryptionkey@0"
I20260926 11:29:45.057911 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d10bca4433d845d992b9ae98c0841faf"
format_stamp: "Formatted at 2026-09-26 11:29:45 on dist-test-slave-mrck"
server_key: "b08ac860883f67de910878bfb255bbee"
server_key_iv: "d34404d3573db53161d21f4c11b3b09e"
server_key_version: "encryptionkey@0"
I20260926 11:29:45.061543 640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 11:29:45.063943 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:45.065182 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:29:45.065320 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "d10bca4433d845d992b9ae98c0841faf"
format_stamp: "Formatted at 2026-09-26 11:29:45 on dist-test-slave-mrck"
server_key: "b08ac860883f67de910878bfb255bbee"
server_key_iv: "d34404d3573db53161d21f4c11b3b09e"
server_key_version: "encryptionkey@0"
I20260926 11:29:45.065438 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:29:45.077340 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:29:45.080811 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:29:45.081022 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:29:45.081619 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:29:45.082481 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:29:45.082559 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:45.082631 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:29:45.082675 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:45.092255 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39181
I20260926 11:29:45.092288 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39181 every 8 connection(s)
I20260926 11:29:45.093140 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:29:45.102438 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 640
I20260926 11:29:45.102540 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-1/wal/instance
Sep 26 11:29:45 dist-test-slave-mrck krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:29:45.102833 392 external_mini_cluster.cc:1428] Setting key 9aa0e24aa2154df4bb225295987f91c4
I20260926 11:29:45.107709 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:45.094913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54179 (local address 127.0.98.62:35449)
0926 11:29:45.095075 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:45.095080 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:45.095967 (+ 887us) server_negotiation.cc:407] Connection header received
0926 11:29:45.097286 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:45.097289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:45.097348 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:45.097446 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:45.098687 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:45.099200 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:45.100340 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:45.100577 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:45.100661 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:45.103399 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:45.103403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:45.103405 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:45.103436 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:45.105322 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:45.105864 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:45.105871 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:45.105876 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:45.105999 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:45.106435 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:45.106438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:45.106440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:45.106616 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:45.106693 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:45.107414 (+ 721us) server_negotiation.cc:299] Negotiation successful
0926 11:29:45.107542 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 11:29:45.108603 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:45.095324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35449 (local address 127.0.98.2:54179)
0926 11:29:45.095826 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:45.095867 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:45.097067 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:45.097659 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:45.097667 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:45.098062 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:29:45.098543 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:45.098558 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:45.099306 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:45.099308 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:45.100225 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:45.100234 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:45.100731 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:45.100734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:45.100893 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:45.101527 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:29:45.101545 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:29:45.103174 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:29:45.105452 (+ 2278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:45.105457 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:45.105470 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:45.105762 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:45.106111 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:45.106117 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:45.106120 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:45.106331 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:45.106720 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:29:45.106730 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:29:45.107101 (+ 371us) client_negotiation.cc:769] Sending connection context
0926 11:29:45.107329 (+ 228us) client_negotiation.cc:240] Negotiation successful
0926 11:29:45.107672 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":98,"threads_started":1}
I20260926 11:29:45.109834 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35449
I20260926 11:29:45.110113 769 heartbeater.cc:464] Registering TS with master...
I20260926 11:29:45.110755 769 heartbeater.cc:511] Master 127.0.98.62:35449 requested a full tablet report, sending...
I20260926 11:29:45.111924 444 ts_manager.cc:194] Registered new tserver with Master: d10bca4433d845d992b9ae98c0841faf (127.0.98.2:39181)
I20260926 11:29:45.112514 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54179
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:29:45.129062 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35449
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:29:45.240705 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:29:45.240965 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:29:45.241010 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:29:45.245177 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:29:45.245230 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:29:45.245288 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:29:45.250494 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35449
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:29:45.251453 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:29:45.252399 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:29:45.261716 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:29:45.261730 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:29:45.261723 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:29:45.262113 774 server_base.cc:1034] running on GCE node
I20260926 11:29:45.262560 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:29:45.263181 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:29:45.264447 774 hybrid_clock.cc:648] HybridClock initialized: now 1790422185264432 us; error 39 us; skew 500 ppm
Sep 26 11:29:45 dist-test-slave-mrck krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:29:45.267225 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:29:45.268149 774 webserver.cc:460] Webserver started at http://127.0.98.3:36609/ using document root <none> and password file <none>
I20260926 11:29:45.268750 774 fs_manager.cc:359] Metadata directory not provided
I20260926 11:29:45.268836 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:29:45.269059 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:29:45.271153 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-2/data/instance:
uuid: "cbb3277e6bed40b2abd721675ad88328"
format_stamp: "Formatted at 2026-09-26 11:29:45 on dist-test-slave-mrck"
server_key: "d44c8396fb128b2e30d722954714bb49"
server_key_iv: "807ee9fcaddd0c814ab40e39879c8a92"
server_key_version: "encryptionkey@0"
I20260926 11:29:45.271832 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cbb3277e6bed40b2abd721675ad88328"
format_stamp: "Formatted at 2026-09-26 11:29:45 on dist-test-slave-mrck"
server_key: "d44c8396fb128b2e30d722954714bb49"
server_key_iv: "807ee9fcaddd0c814ab40e39879c8a92"
server_key_version: "encryptionkey@0"
I20260926 11:29:45.275885 774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:29:45.278214 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:45.279345 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:29:45.279485 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "cbb3277e6bed40b2abd721675ad88328"
format_stamp: "Formatted at 2026-09-26 11:29:45 on dist-test-slave-mrck"
server_key: "d44c8396fb128b2e30d722954714bb49"
server_key_iv: "807ee9fcaddd0c814ab40e39879c8a92"
server_key_version: "encryptionkey@0"
I20260926 11:29:45.279597 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:29:45.291963 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:29:45.295042 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:29:45.295239 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:29:45.295879 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:29:45.296696 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:29:45.296768 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:45.296839 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:29:45.296882 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:45.306449 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43287
I20260926 11:29:45.306463 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43287 every 8 connection(s)
I20260926 11:29:45.307319 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:29:45.315474 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 774
I20260926 11:29:45.315593 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790422180210734-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:29:45.315896 392 external_mini_cluster.cc:1428] Setting key fe66a9bcd138a1041afd08bf6d3e9163
Sep 26 11:29:45 dist-test-slave-mrck krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:29:45.321054 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:45.309070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47617 (local address 127.0.98.62:35449)
0926 11:29:45.309213 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:45.309217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:45.310020 (+ 803us) server_negotiation.cc:407] Connection header received
0926 11:29:45.311340 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:45.311344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:45.311400 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:45.311513 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:45.312751 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:45.313284 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:45.314412 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:45.314647 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:45.314755 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:45.317552 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:45.317558 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:45.317561 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:45.317589 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:45.319128 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:45.319642 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:45.319645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:45.319646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:45.319719 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:45.320079 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:45.320081 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:45.320082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:45.320252 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:45.320328 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:45.320825 (+ 497us) server_negotiation.cc:299] Negotiation successful
0926 11:29:45.320925 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 11:29:45.321854 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:45.309445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35449 (local address 127.0.98.3:47617)
0926 11:29:45.309859 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:45.309902 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:45.311129 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:45.311701 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:45.311708 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:45.312119 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:29:45.312603 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:45.312616 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:45.313411 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:45.313417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:45.314286 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:45.314295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:45.314804 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:45.314808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:45.314971 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:45.315539 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:29:45.315563 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:29:45.317413 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:29:45.319248 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:45.319252 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:45.319266 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:45.319549 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:45.319863 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:45.319865 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:45.319867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:45.320000 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:45.320343 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:29:45.320348 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:29:45.320601 (+ 253us) client_negotiation.cc:769] Sending connection context
0926 11:29:45.320761 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 11:29:45.321013 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":94,"threads_started":1}
I20260926 11:29:45.323016 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35449
I20260926 11:29:45.323231 903 heartbeater.cc:464] Registering TS with master...
I20260926 11:29:45.323681 903 heartbeater.cc:511] Master 127.0.98.62:35449 requested a full tablet report, sending...
I20260926 11:29:45.324713 444 ts_manager.cc:194] Registered new tserver with Master: cbb3277e6bed40b2abd721675ad88328 (127.0.98.3:43287)
I20260926 11:29:45.325314 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47617
I20260926 11:29:45.330392 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 11:29:45 dist-test-slave-mrck krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422185, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 11:29:45 dist-test-slave-mrck krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422185, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 11:29:45.356781 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:45.347692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42158 (local address 127.0.98.1:35001)
0926 11:29:45.348047 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:45.348060 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:45.348094 (+ 34us) server_negotiation.cc:407] Connection header received
0926 11:29:45.348159 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:45.348164 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:45.348356 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:45.348475 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:45.349131 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:45.349892 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:45.350682 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:45.351030 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:45.351110 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:45.353469 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:45.353477 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:45.353490 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:45.353519 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:45.355022 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:45.355469 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:45.355474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:45.355475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:45.355565 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:45.355867 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:45.355870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:45.355871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:45.356100 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:45.356277 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:45.356418 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 11:29:45.356588 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":127,"threads_started":1}
W20260926 11:29:45.357760 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42158
Sep 26 11:29:45 dist-test-slave-mrck krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 11:29:45 dist-test-slave-mrck krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 11:29:45.383095 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:45.375054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42162 (local address 127.0.98.1:35001)
0926 11:29:45.375199 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:45.375205 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:45.375355 (+ 150us) server_negotiation.cc:407] Connection header received
0926 11:29:45.375531 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:45.375533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:45.375575 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:45.375662 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:45.376383 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:45.376901 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:45.377721 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:45.377952 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:45.378065 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:45.380332 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:45.380338 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:45.380344 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:45.380368 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:45.381521 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:45.381910 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:45.381915 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:45.381917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:45.381991 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:45.382246 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:45.382250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:45.382252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:45.382412 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:45.382548 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:45.382773 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 11:29:45.382919 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 11:29:45.384151 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 506
I20260926 11:29:45.389719 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 640
I20260926 11:29:45.397418 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 774
I20260926 11:29:45.403059 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 412
2026-09-26T11:29:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5108 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:29:45 dist-test-slave-mrck krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:29:45 dist-test-slave-mrck krb5kdc[917](info): set up 2 sockets
Sep 26 11:29:45 dist-test-slave-mrck krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260926 11:29:46.427469 392 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:29:46 dist-test-slave-mrck krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:29:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:29:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:29:46.542673 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40755
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40755
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:29:46.649720 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:29:46.650107 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:29:46.650166 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:29:46.654098 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:29:46.654151 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:29:46.654170 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:29:46.654186 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:29:46.654201 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:29:46.659337 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:29:46.660341 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:29:46.661267 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:29:46.669530 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:29:46.669519 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:29:46.669698 930 server_base.cc:1034] running on GCE node
W20260926 11:29:46.669756 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:29:46.670315 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:29:46.671396 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:29:46.672578 930 hybrid_clock.cc:648] HybridClock initialized: now 1790422186672563 us; error 34 us; skew 500 ppm
Sep 26 11:29:46 dist-test-slave-mrck krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:29:46.675112 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:29:46.676051 930 webserver.cc:460] Webserver started at http://127.0.98.62:43751/ using document root <none> and password file <none>
I20260926 11:29:46.676589 930 fs_manager.cc:359] Metadata directory not provided
I20260926 11:29:46.676674 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:29:46.676895 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:29:46.678973 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "7c47a54db24b4deda625d07789c3d9e5"
format_stamp: "Formatted at 2026-09-26 11:29:46 on dist-test-slave-mrck"
server_key: "1eacbabd70373c4f67d6f60d347f9240"
server_key_iv: "d8e57c7ddb1ac1dedee3b2f8084e4c90"
server_key_version: "encryptionkey@0"
I20260926 11:29:46.679517 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "7c47a54db24b4deda625d07789c3d9e5"
format_stamp: "Formatted at 2026-09-26 11:29:46 on dist-test-slave-mrck"
server_key: "1eacbabd70373c4f67d6f60d347f9240"
server_key_iv: "d8e57c7ddb1ac1dedee3b2f8084e4c90"
server_key_version: "encryptionkey@0"
I20260926 11:29:46.683175 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 11:29:46.685413 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:46.686605 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 11:29:46.686736 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "7c47a54db24b4deda625d07789c3d9e5"
format_stamp: "Formatted at 2026-09-26 11:29:46 on dist-test-slave-mrck"
server_key: "1eacbabd70373c4f67d6f60d347f9240"
server_key_iv: "d8e57c7ddb1ac1dedee3b2f8084e4c90"
server_key_version: "encryptionkey@0"
I20260926 11:29:46.686838 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:29:46.704840 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:29:46.707922 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:29:46.708149 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:29:46.716153 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40755
I20260926 11:29:46.716205 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40755 every 8 connection(s)
I20260926 11:29:46.721606 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:29:46.725327 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:29:46.728552 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 930
I20260926 11:29:46.728641 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:29:46.728897 392 external_mini_cluster.cc:1428] Setting key 348690975a1d16654dfcdc271e55b86a
I20260926 11:29:46.731678 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c47a54db24b4deda625d07789c3d9e5: Bootstrap starting.
I20260926 11:29:46.734361 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c47a54db24b4deda625d07789c3d9e5: Neither blocks nor log segments found. Creating new log.
I20260926 11:29:46.735111 998 log.cc:824] T 00000000000000000000000000000000 P 7c47a54db24b4deda625d07789c3d9e5: Log is configured to *not* fsync() on all Append() calls
I20260926 11:29:46.737044 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c47a54db24b4deda625d07789c3d9e5: No bootstrap required, opened a new log
Sep 26 11:29:46 dist-test-slave-mrck krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:29:46.740301 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c47a54db24b4deda625d07789c3d9e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c47a54db24b4deda625d07789c3d9e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40755 } }
I20260926 11:29:46.740558 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c47a54db24b4deda625d07789c3d9e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:29:46.740636 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c47a54db24b4deda625d07789c3d9e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c47a54db24b4deda625d07789c3d9e5, State: Initialized, Role: FOLLOWER
I20260926 11:29:46.741114 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c47a54db24b4deda625d07789c3d9e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c47a54db24b4deda625d07789c3d9e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40755 } }
I20260926 11:29:46.741261 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c47a54db24b4deda625d07789c3d9e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:29:46.741341 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c47a54db24b4deda625d07789c3d9e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:29:46.741460 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c47a54db24b4deda625d07789c3d9e5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:29:46.742311 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c47a54db24b4deda625d07789c3d9e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c47a54db24b4deda625d07789c3d9e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40755 } }
I20260926 11:29:46.742648 998 leader_election.cc:302] T 00000000000000000000000000000000 P 7c47a54db24b4deda625d07789c3d9e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c47a54db24b4deda625d07789c3d9e5; no voters:
I20260926 11:29:46.743026 998 leader_election.cc:288] T 00000000000000000000000000000000 P 7c47a54db24b4deda625d07789c3d9e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:29:46.743144 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c47a54db24b4deda625d07789c3d9e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:29:46.743311 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c47a54db24b4deda625d07789c3d9e5 [term 1 LEADER]: Becoming Leader. State: Replica: 7c47a54db24b4deda625d07789c3d9e5, State: Running, Role: LEADER
I20260926 11:29:46.743598 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c47a54db24b4deda625d07789c3d9e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c47a54db24b4deda625d07789c3d9e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40755 } }
I20260926 11:29:46.744014 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:46.730128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34296 (local address 127.0.98.62:40755)
0926 11:29:46.730542 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:46.730562 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:46.730606 (+ 44us) server_negotiation.cc:407] Connection header received
0926 11:29:46.731425 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:46.731448 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:46.731892 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:46.732258 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:46.732919 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:46.734351 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:46.735285 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:46.735813 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:46.735999 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:46.738464 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:46.738469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:46.738481 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:46.738506 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:46.741010 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:46.741517 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:46.741522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:46.741526 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:46.741664 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:46.741952 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:46.741957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:46.741962 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:46.742336 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:46.742545 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:46.742784 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 11:29:46.743058 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":115,"threads_started":1}
I20260926 11:29:46.744297 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c47a54db24b4deda625d07789c3d9e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:29:46.744766 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c47a54db24b4deda625d07789c3d9e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c47a54db24b4deda625d07789c3d9e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c47a54db24b4deda625d07789c3d9e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40755 } } }
I20260926 11:29:46.744892 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c47a54db24b4deda625d07789c3d9e5 [sys.catalog]: This master's current role is: LEADER
I20260926 11:29:46.745091 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c47a54db24b4deda625d07789c3d9e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c47a54db24b4deda625d07789c3d9e5. Latest consensus state: current_term: 1 leader_uuid: "7c47a54db24b4deda625d07789c3d9e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c47a54db24b4deda625d07789c3d9e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40755 } } }
I20260926 11:29:46.745189 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c47a54db24b4deda625d07789c3d9e5 [sys.catalog]: This master's current role is: LEADER
I20260926 11:29:46.746716 1007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:29:46.749295 1007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:29:46.754693 1007 catalog_manager.cc:1347] Generated new cluster ID: ff6afadd70544d15b66643216cf56b8d
I20260926 11:29:46.754787 1007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:29:46.762979 1007 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:29:46.763824 1007 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:29:46.769302 1007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c47a54db24b4deda625d07789c3d9e5: Generated new TSK 0
I20260926 11:29:46.769930 1007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:29:46.802414 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40755
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:29:46.910360 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:29:46.910620 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:29:46.910665 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:29:46.914654 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:29:46.914714 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:29:46.914772 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:29:46.919982 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40755
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:29:46.920996 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:29:46.922040 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:29:46.931634 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:29:46.931638 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:29:46.931638 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:29:46.931782 1024 server_base.cc:1034] running on GCE node
I20260926 11:29:46.932439 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:29:46.933120 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:29:46.934319 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790422186934299 us; error 30 us; skew 500 ppm
Sep 26 11:29:46 dist-test-slave-mrck krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:29:46.937129 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:29:46.937996 1024 webserver.cc:460] Webserver started at http://127.0.98.1:32979/ using document root <none> and password file <none>
I20260926 11:29:46.938526 1024 fs_manager.cc:359] Metadata directory not provided
I20260926 11:29:46.938612 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:29:46.938830 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:29:46.940819 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-0/data/instance:
uuid: "4203f8228c574ba688c7ecf4c0cf63a8"
format_stamp: "Formatted at 2026-09-26 11:29:46 on dist-test-slave-mrck"
server_key: "bec6940351e833d8517f61b953f87468"
server_key_iv: "6c4367b916ca64e120806d8f2b127926"
server_key_version: "encryptionkey@0"
I20260926 11:29:46.941360 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4203f8228c574ba688c7ecf4c0cf63a8"
format_stamp: "Formatted at 2026-09-26 11:29:46 on dist-test-slave-mrck"
server_key: "bec6940351e833d8517f61b953f87468"
server_key_iv: "6c4367b916ca64e120806d8f2b127926"
server_key_version: "encryptionkey@0"
I20260926 11:29:46.944891 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260926 11:29:46.947324 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:46.948554 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 11:29:46.948684 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "4203f8228c574ba688c7ecf4c0cf63a8"
format_stamp: "Formatted at 2026-09-26 11:29:46 on dist-test-slave-mrck"
server_key: "bec6940351e833d8517f61b953f87468"
server_key_iv: "6c4367b916ca64e120806d8f2b127926"
server_key_version: "encryptionkey@0"
I20260926 11:29:46.948807 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:29:46.960140 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:29:46.963369 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:29:46.963574 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:29:46.964186 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:29:46.964975 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:29:46.965046 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:46.965111 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:29:46.965155 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:46.973747 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40861
I20260926 11:29:46.973766 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40861 every 8 connection(s)
I20260926 11:29:46.974586 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:29:46.978073 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 1024
I20260926 11:29:46.978173 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:29:46.978420 392 external_mini_cluster.cc:1428] Setting key 94ecbe297bc219f27b554b9379d25e42
Sep 26 11:29:46 dist-test-slave-mrck krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 11:29:46.989658 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:46.976303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54527 (local address 127.0.98.62:40755)
0926 11:29:46.976437 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:46.976440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:46.977159 (+ 719us) server_negotiation.cc:407] Connection header received
0926 11:29:46.978720 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:46.978723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:46.978795 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:46.978877 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:46.980263 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:46.980790 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:46.982042 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:46.982271 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:46.982381 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:46.985226 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:46.985233 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:46.985239 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:46.985270 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:46.986912 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:46.987414 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:46.987418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:46.987420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:46.987539 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:46.988156 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:46.988160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:46.988161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:46.988543 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:46.988666 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:46.989315 (+ 649us) server_negotiation.cc:299] Negotiation successful
0926 11:29:46.989473 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 11:29:46.990598 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:46.976609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40755 (local address 127.0.98.1:54527)
0926 11:29:46.977011 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:46.977050 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:46.978485 (+ 1435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:46.979075 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:46.979085 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:46.979430 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:29:46.980113 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:46.980132 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:46.980907 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:46.980914 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:46.981913 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:46.981921 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:46.982412 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:46.982416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:46.982627 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:46.983191 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:29:46.983220 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:29:46.985049 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:29:46.987045 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:46.987050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:46.987060 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:46.987294 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:46.987706 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:46.987710 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:46.987711 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:46.988009 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:46.988690 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:29:46.988697 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:29:46.989042 (+ 345us) client_negotiation.cc:769] Sending connection context
0926 11:29:46.989261 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 11:29:46.989636 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":108,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 11:29:46.991964 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40755
I20260926 11:29:46.992219 1153 heartbeater.cc:464] Registering TS with master...
I20260926 11:29:46.992851 1153 heartbeater.cc:511] Master 127.0.98.62:40755 requested a full tablet report, sending...
I20260926 11:29:46.994427 962 ts_manager.cc:194] Registered new tserver with Master: 4203f8228c574ba688c7ecf4c0cf63a8 (127.0.98.1:40861)
I20260926 11:29:46.995532 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54527
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:29:47.006801 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40755
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:29:47.115651 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:29:47.115943 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:29:47.115988 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:29:47.119910 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:29:47.119964 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:29:47.120039 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:29:47.125221 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40755
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:29:47.126205 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:29:47.127130 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:29:47.136420 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:29:47.136451 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:29:47.136420 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:29:47.136693 1158 server_base.cc:1034] running on GCE node
I20260926 11:29:47.137096 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:29:47.137652 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:29:47.138818 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790422187138796 us; error 29 us; skew 500 ppm
Sep 26 11:29:47 dist-test-slave-mrck krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:29:47.141673 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:29:47.142602 1158 webserver.cc:460] Webserver started at http://127.0.98.2:43545/ using document root <none> and password file <none>
I20260926 11:29:47.143155 1158 fs_manager.cc:359] Metadata directory not provided
I20260926 11:29:47.143246 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:29:47.143514 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:29:47.145910 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-1/data/instance:
uuid: "d7cd4432e8774ea5a6ccd3184c9efb10"
format_stamp: "Formatted at 2026-09-26 11:29:47 on dist-test-slave-mrck"
server_key: "24b3955a94458975b6e8ade23274bd47"
server_key_iv: "3f89428cf5b6cb11bc5f0ab7095b5740"
server_key_version: "encryptionkey@0"
I20260926 11:29:47.146508 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d7cd4432e8774ea5a6ccd3184c9efb10"
format_stamp: "Formatted at 2026-09-26 11:29:47 on dist-test-slave-mrck"
server_key: "24b3955a94458975b6e8ade23274bd47"
server_key_iv: "3f89428cf5b6cb11bc5f0ab7095b5740"
server_key_version: "encryptionkey@0"
I20260926 11:29:47.150557 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 11:29:47.153263 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:47.154433 1158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 11:29:47.154579 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "d7cd4432e8774ea5a6ccd3184c9efb10"
format_stamp: "Formatted at 2026-09-26 11:29:47 on dist-test-slave-mrck"
server_key: "24b3955a94458975b6e8ade23274bd47"
server_key_iv: "3f89428cf5b6cb11bc5f0ab7095b5740"
server_key_version: "encryptionkey@0"
I20260926 11:29:47.154691 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:29:47.175479 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:29:47.178725 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:29:47.178960 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:29:47.179584 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:29:47.180483 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:29:47.180562 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:47.180634 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:29:47.180685 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:47.190026 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43215
I20260926 11:29:47.190093 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43215 every 8 connection(s)
I20260926 11:29:47.190969 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:29:47.193087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 1158
I20260926 11:29:47.193276 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:29:47.193560 392 external_mini_cluster.cc:1428] Setting key 0e99bf70be6fa35f9cc287c8185e976d
Sep 26 11:29:47 dist-test-slave-mrck krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 11:29:47.207221 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:47.192902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49539 (local address 127.0.98.62:40755)
0926 11:29:47.193084 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:47.193089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:47.194700 (+ 1611us) server_negotiation.cc:407] Connection header received
0926 11:29:47.196150 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:47.196156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:47.196219 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:47.196317 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:47.197549 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.198447 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.199673 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.200048 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.200161 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:47.203335 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:47.203340 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:47.203343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:47.203371 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:47.204965 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:47.205488 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:47.205491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:47.205492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:47.205590 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:47.205902 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:47.205904 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:47.205905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:47.206167 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:47.206271 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:47.206972 (+ 701us) server_negotiation.cc:299] Negotiation successful
0926 11:29:47.207088 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 11:29:47.208403 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:47.193364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40755 (local address 127.0.98.2:49539)
0926 11:29:47.194509 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:47.194567 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:47.195931 (+ 1364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:47.196526 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:47.196535 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:47.196898 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:29:47.197395 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:47.197410 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.198612 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.198615 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:47.199550 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:47.199560 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.200236 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.200243 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:47.200515 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:47.201189 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:29:47.201218 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:29:47.203145 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:29:47.205083 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:47.205088 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:47.205103 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:47.205361 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:47.205676 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:47.205679 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:47.205680 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:47.205819 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:47.206294 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:29:47.206300 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:29:47.206651 (+ 351us) client_negotiation.cc:769] Sending connection context
0926 11:29:47.206908 (+ 257us) client_negotiation.cc:240] Negotiation successful
0926 11:29:47.207308 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"thread_start_us":140,"threads_started":1}
I20260926 11:29:47.209692 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40755
I20260926 11:29:47.209961 1287 heartbeater.cc:464] Registering TS with master...
I20260926 11:29:47.210618 1287 heartbeater.cc:511] Master 127.0.98.62:40755 requested a full tablet report, sending...
I20260926 11:29:47.211874 962 ts_manager.cc:194] Registered new tserver with Master: d7cd4432e8774ea5a6ccd3184c9efb10 (127.0.98.2:43215)
I20260926 11:29:47.212459 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49539
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:29:47.222466 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40755
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:29:47.332985 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:29:47.333293 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:29:47.333380 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:29:47.337615 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:29:47.337713 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:29:47.337803 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:29:47.343376 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40755
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:29:47.344573 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:29:47.345741 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:29:47.355356 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:29:47.355466 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:29:47.355532 1292 server_base.cc:1034] running on GCE node
W20260926 11:29:47.355356 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:29:47.356129 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:29:47.356724 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:29:47.357880 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790422187357859 us; error 32 us; skew 500 ppm
Sep 26 11:29:47 dist-test-slave-mrck krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:29:47.360647 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:29:47.361588 1292 webserver.cc:460] Webserver started at http://127.0.98.3:36869/ using document root <none> and password file <none>
I20260926 11:29:47.362134 1292 fs_manager.cc:359] Metadata directory not provided
I20260926 11:29:47.362192 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:29:47.362422 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:29:47.364526 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-2/data/instance:
uuid: "014ab61c549a4a02a34bede119434fda"
format_stamp: "Formatted at 2026-09-26 11:29:47 on dist-test-slave-mrck"
server_key: "756d3caa814df0882fc6e529ae7ff178"
server_key_iv: "f54baf701bbe53ee0074968f49bf3c66"
server_key_version: "encryptionkey@0"
I20260926 11:29:47.365082 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "014ab61c549a4a02a34bede119434fda"
format_stamp: "Formatted at 2026-09-26 11:29:47 on dist-test-slave-mrck"
server_key: "756d3caa814df0882fc6e529ae7ff178"
server_key_iv: "f54baf701bbe53ee0074968f49bf3c66"
server_key_version: "encryptionkey@0"
I20260926 11:29:47.368690 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 11:29:47.371063 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:47.372454 1292 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:29:47.372591 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "014ab61c549a4a02a34bede119434fda"
format_stamp: "Formatted at 2026-09-26 11:29:47 on dist-test-slave-mrck"
server_key: "756d3caa814df0882fc6e529ae7ff178"
server_key_iv: "f54baf701bbe53ee0074968f49bf3c66"
server_key_version: "encryptionkey@0"
I20260926 11:29:47.372699 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:29:47.391501 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:29:47.394467 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:29:47.394675 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:29:47.395259 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:29:47.396097 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:29:47.396165 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:47.396236 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:29:47.396286 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:47.404882 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37523
I20260926 11:29:47.404901 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37523 every 8 connection(s)
I20260926 11:29:47.405781 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:29:47.408663 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 1292
I20260926 11:29:47.408777 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790422180210734-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:29:47.409039 392 external_mini_cluster.cc:1428] Setting key 5f471680ab67daa205eccf038455db52
Sep 26 11:29:47 dist-test-slave-mrck krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:29:47.418877 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:47.407542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58371 (local address 127.0.98.62:40755)
0926 11:29:47.407705 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:47.407710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:47.408531 (+ 821us) server_negotiation.cc:407] Connection header received
0926 11:29:47.409959 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:47.409962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:47.410030 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:47.410140 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:47.411255 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.411856 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.412984 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.413213 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.413326 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:47.415608 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:47.415611 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:47.415614 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:47.415644 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:47.416888 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:47.417377 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:47.417381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:47.417383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:47.417465 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:47.417798 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:47.417802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:47.417803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:47.417988 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:47.418097 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:47.418590 (+ 493us) server_negotiation.cc:299] Negotiation successful
0926 11:29:47.418709 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 11:29:47.419634 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:47.407949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40755 (local address 127.0.98.3:58371)
0926 11:29:47.408387 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:47.408428 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:47.409757 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:47.410310 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:47.410318 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:47.410661 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:29:47.411119 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:47.411132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.411973 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.411977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:47.412857 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:47.412866 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.413356 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.413360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:47.413533 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:47.414070 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:29:47.414085 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:29:47.415436 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:29:47.417001 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:47.417009 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:47.417025 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:47.417274 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:47.417563 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:47.417566 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:47.417568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:47.417697 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:47.418104 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:29:47.418112 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:29:47.418349 (+ 237us) client_negotiation.cc:769] Sending connection context
0926 11:29:47.418547 (+ 198us) client_negotiation.cc:240] Negotiation successful
0926 11:29:47.418855 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":122,"threads_started":1}
I20260926 11:29:47.420701 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40755
I20260926 11:29:47.420912 1421 heartbeater.cc:464] Registering TS with master...
I20260926 11:29:47.421339 1421 heartbeater.cc:511] Master 127.0.98.62:40755 requested a full tablet report, sending...
I20260926 11:29:47.422299 961 ts_manager.cc:194] Registered new tserver with Master: 014ab61c549a4a02a34bede119434fda (127.0.98.3:37523)
I20260926 11:29:47.422847 961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58371
I20260926 11:29:47.434041 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:29:47.444051 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:47.437328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34312 (local address 127.0.98.62:40755)
0926 11:29:47.437499 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:47.437502 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:47.437560 (+ 58us) server_negotiation.cc:407] Connection header received
0926 11:29:47.437674 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:47.437676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:47.437713 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:47.437776 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:47.438403 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.438921 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.439672 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.439927 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.440003 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:47.441200 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:47.441202 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:47.441204 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:47.441224 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:47.442648 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:47.443000 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:47.443002 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:47.443004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:47.443070 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:47.443278 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:47.443281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:47.443282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:47.443483 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:47.443584 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:47.443774 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 11:29:47.443868 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 11:29:47.448731 961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34312:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:29:47.451880 961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:29:47.465747 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:47.460133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43215 (local address 127.0.0.1:51526)
0926 11:29:47.461481 (+ 1348us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:47.461491 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:47.461594 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:47.462035 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:47.462038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:47.462055 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:29:47.462134 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:47.462139 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.463195 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.463201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:47.464386 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:47.464394 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.465202 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.465206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:47.465315 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:47.465356 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 11:29:47.465468 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 11:29:47.465598 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1286,"mutex_wait_us":43,"thread_start_us":87,"threads_started":1}
I20260926 11:29:47.465755 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:47.460574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51526 (local address 127.0.98.2:43215)
0926 11:29:47.461352 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:47.461363 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:47.461575 (+ 212us) server_negotiation.cc:407] Connection header received
0926 11:29:47.461702 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:47.461707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:47.461889 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:47.462043 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:29:47.462226 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.463071 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.464509 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.465090 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.465200 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:47.465341 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:47.465508 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 11:29:47.465602 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":105,"threads_started":1}
I20260926 11:29:47.466460 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:47.459797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40861 (local address 127.0.0.1:57484)
0926 11:29:47.460437 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:47.460488 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:47.460613 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:47.462370 (+ 1757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:47.462373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:47.462389 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:29:47.462499 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:47.462504 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.463854 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.463857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:47.464968 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:47.464976 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.466065 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.466068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:47.466151 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:47.466172 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 11:29:47.466257 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 11:29:47.466326 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":1,"thread_start_us":59,"threads_started":1}
I20260926 11:29:47.466529 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:47.459951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57484 (local address 127.0.98.1:40861)
0926 11:29:47.461812 (+ 1861us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:47.461826 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:47.461846 (+ 20us) server_negotiation.cc:407] Connection header received
0926 11:29:47.461913 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:47.461918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:47.462081 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:47.462212 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:29:47.462622 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.463684 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.465103 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.465954 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.466056 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:47.466188 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:47.466284 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 11:29:47.466368 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1772,"thread_start_us":126,"threads_started":1}
I20260926 11:29:47.467067 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:47.459889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37523 (local address 127.0.0.1:41728)
0926 11:29:47.461002 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:47.461021 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:47.461096 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:47.461618 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:47.461624 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:47.461646 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:29:47.461786 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:47.461793 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.463095 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.463099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:47.464557 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:47.464566 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.466187 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.466195 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:47.466287 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:47.466309 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 11:29:47.466377 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 11:29:47.466443 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1056,"mutex_wait_us":59,"thread_start_us":106,"threads_started":1}
I20260926 11:29:47.467670 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:47.460574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41728 (local address 127.0.98.3:37523)
0926 11:29:47.460981 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:47.460993 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:47.461082 (+ 89us) server_negotiation.cc:407] Connection header received
0926 11:29:47.461246 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:47.461254 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:47.461461 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:47.461632 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:29:47.461918 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.462949 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.464685 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.465530 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.465624 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:47.465753 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:47.467427 (+ 1674us) server_negotiation.cc:299] Negotiation successful
0926 11:29:47.467532 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":126,"threads_started":1}
I20260926 11:29:47.468931 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 8d3cf18588ac4b92bb8f9c8102633346 (DEFAULT_TABLE table=test-table [id=ec226c83615f440e954959dac5bdace5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:29:47.469018 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 8d3cf18588ac4b92bb8f9c8102633346 (DEFAULT_TABLE table=test-table [id=ec226c83615f440e954959dac5bdace5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:29:47.469915 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d3cf18588ac4b92bb8f9c8102633346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:29:47.469995 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d3cf18588ac4b92bb8f9c8102633346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:29:47.470073 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 8d3cf18588ac4b92bb8f9c8102633346 (DEFAULT_TABLE table=test-table [id=ec226c83615f440e954959dac5bdace5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:29:47.471010 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d3cf18588ac4b92bb8f9c8102633346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:29:47.475915 1438 tablet_bootstrap.cc:492] T 8d3cf18588ac4b92bb8f9c8102633346 P d7cd4432e8774ea5a6ccd3184c9efb10: Bootstrap starting.
I20260926 11:29:47.477545 1439 tablet_bootstrap.cc:492] T 8d3cf18588ac4b92bb8f9c8102633346 P 4203f8228c574ba688c7ecf4c0cf63a8: Bootstrap starting.
I20260926 11:29:47.479045 1438 tablet_bootstrap.cc:654] T 8d3cf18588ac4b92bb8f9c8102633346 P d7cd4432e8774ea5a6ccd3184c9efb10: Neither blocks nor log segments found. Creating new log.
I20260926 11:29:47.479063 1439 tablet_bootstrap.cc:654] T 8d3cf18588ac4b92bb8f9c8102633346 P 4203f8228c574ba688c7ecf4c0cf63a8: Neither blocks nor log segments found. Creating new log.
I20260926 11:29:47.479720 1438 log.cc:824] T 8d3cf18588ac4b92bb8f9c8102633346 P d7cd4432e8774ea5a6ccd3184c9efb10: Log is configured to *not* fsync() on all Append() calls
I20260926 11:29:47.480051 1440 tablet_bootstrap.cc:492] T 8d3cf18588ac4b92bb8f9c8102633346 P 014ab61c549a4a02a34bede119434fda: Bootstrap starting.
I20260926 11:29:47.480142 1439 log.cc:824] T 8d3cf18588ac4b92bb8f9c8102633346 P 4203f8228c574ba688c7ecf4c0cf63a8: Log is configured to *not* fsync() on all Append() calls
I20260926 11:29:47.481415 1438 tablet_bootstrap.cc:492] T 8d3cf18588ac4b92bb8f9c8102633346 P d7cd4432e8774ea5a6ccd3184c9efb10: No bootstrap required, opened a new log
I20260926 11:29:47.481611 1438 ts_tablet_manager.cc:1397] T 8d3cf18588ac4b92bb8f9c8102633346 P d7cd4432e8774ea5a6ccd3184c9efb10: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 11:29:47.481621 1440 tablet_bootstrap.cc:654] T 8d3cf18588ac4b92bb8f9c8102633346 P 014ab61c549a4a02a34bede119434fda: Neither blocks nor log segments found. Creating new log.
I20260926 11:29:47.482780 1440 log.cc:824] T 8d3cf18588ac4b92bb8f9c8102633346 P 014ab61c549a4a02a34bede119434fda: Log is configured to *not* fsync() on all Append() calls
I20260926 11:29:47.484337 1440 tablet_bootstrap.cc:492] T 8d3cf18588ac4b92bb8f9c8102633346 P 014ab61c549a4a02a34bede119434fda: No bootstrap required, opened a new log
I20260926 11:29:47.484512 1440 ts_tablet_manager.cc:1397] T 8d3cf18588ac4b92bb8f9c8102633346 P 014ab61c549a4a02a34bede119434fda: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 11:29:47.485167 1438 raft_consensus.cc:348] T 8d3cf18588ac4b92bb8f9c8102633346 P d7cd4432e8774ea5a6ccd3184c9efb10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4203f8228c574ba688c7ecf4c0cf63a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40861 } } peers { permanent_uuid: "014ab61c549a4a02a34bede119434fda" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37523 } } peers { permanent_uuid: "d7cd4432e8774ea5a6ccd3184c9efb10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } }
I20260926 11:29:47.485394 1438 raft_consensus.cc:374] T 8d3cf18588ac4b92bb8f9c8102633346 P d7cd4432e8774ea5a6ccd3184c9efb10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:29:47.485445 1438 raft_consensus.cc:729] T 8d3cf18588ac4b92bb8f9c8102633346 P d7cd4432e8774ea5a6ccd3184c9efb10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7cd4432e8774ea5a6ccd3184c9efb10, State: Initialized, Role: FOLLOWER
I20260926 11:29:47.485857 1438 consensus_queue.cc:260] T 8d3cf18588ac4b92bb8f9c8102633346 P d7cd4432e8774ea5a6ccd3184c9efb10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4203f8228c574ba688c7ecf4c0cf63a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40861 } } peers { permanent_uuid: "014ab61c549a4a02a34bede119434fda" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37523 } } peers { permanent_uuid: "d7cd4432e8774ea5a6ccd3184c9efb10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } }
I20260926 11:29:47.486521 1287 heartbeater.cc:503] Master 127.0.98.62:40755 was elected leader, sending a full tablet report...
I20260926 11:29:47.486748 1438 ts_tablet_manager.cc:1428] T 8d3cf18588ac4b92bb8f9c8102633346 P d7cd4432e8774ea5a6ccd3184c9efb10: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 11:29:47.487519 1440 raft_consensus.cc:348] T 8d3cf18588ac4b92bb8f9c8102633346 P 014ab61c549a4a02a34bede119434fda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4203f8228c574ba688c7ecf4c0cf63a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40861 } } peers { permanent_uuid: "014ab61c549a4a02a34bede119434fda" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37523 } } peers { permanent_uuid: "d7cd4432e8774ea5a6ccd3184c9efb10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } }
I20260926 11:29:47.487767 1440 raft_consensus.cc:374] T 8d3cf18588ac4b92bb8f9c8102633346 P 014ab61c549a4a02a34bede119434fda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:29:47.487824 1440 raft_consensus.cc:729] T 8d3cf18588ac4b92bb8f9c8102633346 P 014ab61c549a4a02a34bede119434fda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 014ab61c549a4a02a34bede119434fda, State: Initialized, Role: FOLLOWER
I20260926 11:29:47.488240 1440 consensus_queue.cc:260] T 8d3cf18588ac4b92bb8f9c8102633346 P 014ab61c549a4a02a34bede119434fda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4203f8228c574ba688c7ecf4c0cf63a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40861 } } peers { permanent_uuid: "014ab61c549a4a02a34bede119434fda" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37523 } } peers { permanent_uuid: "d7cd4432e8774ea5a6ccd3184c9efb10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } }
I20260926 11:29:47.488569 1439 tablet_bootstrap.cc:492] T 8d3cf18588ac4b92bb8f9c8102633346 P 4203f8228c574ba688c7ecf4c0cf63a8: No bootstrap required, opened a new log
I20260926 11:29:47.488741 1439 ts_tablet_manager.cc:1397] T 8d3cf18588ac4b92bb8f9c8102633346 P 4203f8228c574ba688c7ecf4c0cf63a8: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260926 11:29:47.488956 1421 heartbeater.cc:503] Master 127.0.98.62:40755 was elected leader, sending a full tablet report...
I20260926 11:29:47.489167 1440 ts_tablet_manager.cc:1428] T 8d3cf18588ac4b92bb8f9c8102633346 P 014ab61c549a4a02a34bede119434fda: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260926 11:29:47.492022 1439 raft_consensus.cc:348] T 8d3cf18588ac4b92bb8f9c8102633346 P 4203f8228c574ba688c7ecf4c0cf63a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4203f8228c574ba688c7ecf4c0cf63a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40861 } } peers { permanent_uuid: "014ab61c549a4a02a34bede119434fda" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37523 } } peers { permanent_uuid: "d7cd4432e8774ea5a6ccd3184c9efb10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } }
I20260926 11:29:47.492238 1439 raft_consensus.cc:374] T 8d3cf18588ac4b92bb8f9c8102633346 P 4203f8228c574ba688c7ecf4c0cf63a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:29:47.492300 1439 raft_consensus.cc:729] T 8d3cf18588ac4b92bb8f9c8102633346 P 4203f8228c574ba688c7ecf4c0cf63a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4203f8228c574ba688c7ecf4c0cf63a8, State: Initialized, Role: FOLLOWER
I20260926 11:29:47.492619 1439 consensus_queue.cc:260] T 8d3cf18588ac4b92bb8f9c8102633346 P 4203f8228c574ba688c7ecf4c0cf63a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4203f8228c574ba688c7ecf4c0cf63a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40861 } } peers { permanent_uuid: "014ab61c549a4a02a34bede119434fda" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37523 } } peers { permanent_uuid: "d7cd4432e8774ea5a6ccd3184c9efb10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } }
I20260926 11:29:47.493142 1153 heartbeater.cc:503] Master 127.0.98.62:40755 was elected leader, sending a full tablet report...
I20260926 11:29:47.493392 1439 ts_tablet_manager.cc:1428] T 8d3cf18588ac4b92bb8f9c8102633346 P 4203f8228c574ba688c7ecf4c0cf63a8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 11:29:47.523551 1445 raft_consensus.cc:482] T 8d3cf18588ac4b92bb8f9c8102633346 P 014ab61c549a4a02a34bede119434fda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:29:47.523780 1445 raft_consensus.cc:504] T 8d3cf18588ac4b92bb8f9c8102633346 P 014ab61c549a4a02a34bede119434fda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4203f8228c574ba688c7ecf4c0cf63a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40861 } } peers { permanent_uuid: "014ab61c549a4a02a34bede119434fda" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37523 } } peers { permanent_uuid: "d7cd4432e8774ea5a6ccd3184c9efb10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } }
I20260926 11:29:47.524926 1445 leader_election.cc:288] T 8d3cf18588ac4b92bb8f9c8102633346 P 014ab61c549a4a02a34bede119434fda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4203f8228c574ba688c7ecf4c0cf63a8 (127.0.98.1:40861), d7cd4432e8774ea5a6ccd3184c9efb10 (127.0.98.2:43215)
I20260926 11:29:47.529944 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:47.525296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50385 (local address 127.0.98.1:40861)
0926 11:29:47.525429 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:47.525434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:47.525476 (+ 42us) server_negotiation.cc:407] Connection header received
0926 11:29:47.525531 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:47.525536 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:47.525591 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:47.525649 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:29:47.526054 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.526841 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.528715 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.529310 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.529417 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:47.529477 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:47.529726 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 11:29:47.529820 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 11:29:47.529944 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:47.525315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39403 (local address 127.0.98.2:43215)
0926 11:29:47.525443 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:47.525450 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:47.525739 (+ 289us) server_negotiation.cc:407] Connection header received
0926 11:29:47.525823 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:47.525825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:47.525863 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:47.525922 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:29:47.526209 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.526918 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.528715 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.529310 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.529417 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:47.529477 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:47.529726 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 11:29:47.529820 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 11:29:47.529987 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:47.525161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40861 (local address 127.0.98.3:50385)
0926 11:29:47.525303 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:47.525318 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:47.525408 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:47.525694 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:47.525699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:47.525742 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:29:47.525872 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:47.525877 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.527002 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.527007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:47.528576 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:47.528589 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.529440 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.529444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:47.529572 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:47.529623 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 11:29:47.529699 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 11:29:47.529845 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":176,"spinlock_wait_cycles":33536}
I20260926 11:29:47.529999 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:47.525180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43215 (local address 127.0.98.3:39403)
0926 11:29:47.525612 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:47.525622 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:47.525743 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:47.525979 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:47.525985 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:47.526006 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:29:47.526104 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:47.526108 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.527029 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.527032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:47.528576 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:47.528589 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.529444 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.529447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:47.529547 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:47.529629 (+ 82us) client_negotiation.cc:769] Sending connection context
0926 11:29:47.529699 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 11:29:47.529840 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":276,"spinlock_wait_cycles":1408,"thread_start_us":112,"threads_started":1}
I20260926 11:29:47.530625 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3cf18588ac4b92bb8f9c8102633346" candidate_uuid: "014ab61c549a4a02a34bede119434fda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4203f8228c574ba688c7ecf4c0cf63a8" is_pre_election: true
I20260926 11:29:47.530669 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3cf18588ac4b92bb8f9c8102633346" candidate_uuid: "014ab61c549a4a02a34bede119434fda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cd4432e8774ea5a6ccd3184c9efb10" is_pre_election: true
I20260926 11:29:47.530920 1107 raft_consensus.cc:2454] T 8d3cf18588ac4b92bb8f9c8102633346 P 4203f8228c574ba688c7ecf4c0cf63a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 014ab61c549a4a02a34bede119434fda in term 0.
I20260926 11:29:47.530970 1241 raft_consensus.cc:2454] T 8d3cf18588ac4b92bb8f9c8102633346 P d7cd4432e8774ea5a6ccd3184c9efb10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 014ab61c549a4a02a34bede119434fda in term 0.
I20260926 11:29:47.531453 1308 leader_election.cc:302] T 8d3cf18588ac4b92bb8f9c8102633346 P 014ab61c549a4a02a34bede119434fda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 014ab61c549a4a02a34bede119434fda, 4203f8228c574ba688c7ecf4c0cf63a8, d7cd4432e8774ea5a6ccd3184c9efb10; no voters:
I20260926 11:29:47.531716 1445 raft_consensus.cc:2781] T 8d3cf18588ac4b92bb8f9c8102633346 P 014ab61c549a4a02a34bede119434fda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:29:47.531834 1445 raft_consensus.cc:482] T 8d3cf18588ac4b92bb8f9c8102633346 P 014ab61c549a4a02a34bede119434fda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:29:47.531903 1445 raft_consensus.cc:3037] T 8d3cf18588ac4b92bb8f9c8102633346 P 014ab61c549a4a02a34bede119434fda [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:29:47.532927 1445 raft_consensus.cc:504] T 8d3cf18588ac4b92bb8f9c8102633346 P 014ab61c549a4a02a34bede119434fda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4203f8228c574ba688c7ecf4c0cf63a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40861 } } peers { permanent_uuid: "014ab61c549a4a02a34bede119434fda" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37523 } } peers { permanent_uuid: "d7cd4432e8774ea5a6ccd3184c9efb10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } }
I20260926 11:29:47.533298 1445 leader_election.cc:288] T 8d3cf18588ac4b92bb8f9c8102633346 P 014ab61c549a4a02a34bede119434fda [CANDIDATE]: Term 1 election: Requested vote from peers 4203f8228c574ba688c7ecf4c0cf63a8 (127.0.98.1:40861), d7cd4432e8774ea5a6ccd3184c9efb10 (127.0.98.2:43215)
I20260926 11:29:47.533608 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3cf18588ac4b92bb8f9c8102633346" candidate_uuid: "014ab61c549a4a02a34bede119434fda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4203f8228c574ba688c7ecf4c0cf63a8"
I20260926 11:29:47.533656 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d3cf18588ac4b92bb8f9c8102633346" candidate_uuid: "014ab61c549a4a02a34bede119434fda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cd4432e8774ea5a6ccd3184c9efb10"
I20260926 11:29:47.533715 1107 raft_consensus.cc:3037] T 8d3cf18588ac4b92bb8f9c8102633346 P 4203f8228c574ba688c7ecf4c0cf63a8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:29:47.533788 1241 raft_consensus.cc:3037] T 8d3cf18588ac4b92bb8f9c8102633346 P d7cd4432e8774ea5a6ccd3184c9efb10 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:29:47.534783 1107 raft_consensus.cc:2454] T 8d3cf18588ac4b92bb8f9c8102633346 P 4203f8228c574ba688c7ecf4c0cf63a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 014ab61c549a4a02a34bede119434fda in term 1.
I20260926 11:29:47.534950 1241 raft_consensus.cc:2454] T 8d3cf18588ac4b92bb8f9c8102633346 P d7cd4432e8774ea5a6ccd3184c9efb10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 014ab61c549a4a02a34bede119434fda in term 1.
I20260926 11:29:47.535082 1310 leader_election.cc:302] T 8d3cf18588ac4b92bb8f9c8102633346 P 014ab61c549a4a02a34bede119434fda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 014ab61c549a4a02a34bede119434fda, 4203f8228c574ba688c7ecf4c0cf63a8; no voters:
I20260926 11:29:47.535247 1445 raft_consensus.cc:2781] T 8d3cf18588ac4b92bb8f9c8102633346 P 014ab61c549a4a02a34bede119434fda [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:29:47.535439 1445 raft_consensus.cc:686] T 8d3cf18588ac4b92bb8f9c8102633346 P 014ab61c549a4a02a34bede119434fda [term 1 LEADER]: Becoming Leader. State: Replica: 014ab61c549a4a02a34bede119434fda, State: Running, Role: LEADER
I20260926 11:29:47.535811 1445 consensus_queue.cc:237] T 8d3cf18588ac4b92bb8f9c8102633346 P 014ab61c549a4a02a34bede119434fda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4203f8228c574ba688c7ecf4c0cf63a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40861 } } peers { permanent_uuid: "014ab61c549a4a02a34bede119434fda" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37523 } } peers { permanent_uuid: "d7cd4432e8774ea5a6ccd3184c9efb10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } }
I20260926 11:29:47.538909 962 catalog_manager.cc:5495] T 8d3cf18588ac4b92bb8f9c8102633346 P 014ab61c549a4a02a34bede119434fda reported cstate change: term changed from 0 to 1, leader changed from <none> to 014ab61c549a4a02a34bede119434fda (127.0.98.3). New cstate: current_term: 1 leader_uuid: "014ab61c549a4a02a34bede119434fda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4203f8228c574ba688c7ecf4c0cf63a8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "014ab61c549a4a02a34bede119434fda" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7cd4432e8774ea5a6ccd3184c9efb10" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43215 } health_report { overall_health: UNKNOWN } } }
Sep 26 11:29:47 dist-test-slave-mrck krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422186, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 11:29:47.566891 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:47.557690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57500 (local address 127.0.98.1:40861)
0926 11:29:47.557834 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:47.557839 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:47.558023 (+ 184us) server_negotiation.cc:407] Connection header received
0926 11:29:47.558194 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:47.558196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:47.558237 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:47.558323 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:47.559076 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.559611 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.560944 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.561159 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.561276 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:47.563837 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:47.563849 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:47.563865 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:47.563889 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:47.565388 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:47.565796 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:47.565802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:47.565803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:47.565883 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:47.566107 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:47.566109 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:47.566110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:47.566340 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:47.566507 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:47.566658 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 11:29:47.566756 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 11:29:47 dist-test-slave-mrck krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422187, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 11:29:47 dist-test-slave-mrck krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422187, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 11:29:47.593954 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:47.585125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57510 (local address 127.0.98.1:40861)
0926 11:29:47.585314 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:47.585318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:47.585337 (+ 19us) server_negotiation.cc:407] Connection header received
0926 11:29:47.585486 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:47.585488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:47.585535 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:47.585611 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:47.586309 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.586835 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.587654 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.587954 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.588053 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:47.590613 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:47.590616 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:47.590618 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:47.590642 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:47.592185 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:47.592641 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:47.592644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:47.592645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:47.592725 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:47.593019 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:47.593022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:47.593023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:47.593195 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:47.593294 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:47.593581 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 11:29:47.593738 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
Sep 26 11:29:47 dist-test-slave-mrck krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 11:29:47 dist-test-slave-mrck krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 11:29:47.618726 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:47.610692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57526 (local address 127.0.98.1:40861)
0926 11:29:47.610830 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:47.610836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:47.611020 (+ 184us) server_negotiation.cc:407] Connection header received
0926 11:29:47.611200 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:47.611203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:47.611253 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:47.611338 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:47.612095 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.612611 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.613455 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:47.613669 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:47.613781 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:47.616205 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:47.616208 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:47.616210 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:47.616236 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:47.617426 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:47.617786 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:47.617788 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:47.617790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:47.617856 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:47.618067 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:47.618070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:47.618071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:47.618218 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:47.618318 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:47.618512 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 11:29:47.618607 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 11:29:47.620250 1067 tablet_service.cc:2908] Rejecting scan request for tablet 8d3cf18588ac4b92bb8f9c8102633346: Uninitialized: clean time has not yet been initialized
I20260926 11:29:47.622442 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 1024
I20260926 11:29:47.628310 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 1158
I20260926 11:29:47.634014 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 1292
I20260926 11:29:47.639379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 930
2026-09-26T11:29:47Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2236 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:29:47 dist-test-slave-mrck krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:29:47 dist-test-slave-mrck krb5kdc[1462](info): set up 2 sockets
Sep 26 11:29:47 dist-test-slave-mrck krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260926 11:29:48.663534 392 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:29:48 dist-test-slave-mrck krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:29:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:29:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:29:48.782763 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44061
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:29:48.891964 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:29:48.892294 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:29:48.892367 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:29:48.896405 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:29:48.896513 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 11:29:48.896548 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:29:48.896598 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:29:48.896642 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:29:48.896678 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:29:48.902647 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44061
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:29:48.903825 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:29:48.904862 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:29:48.913311 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:29:48.913354 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:29:48.913311 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:29:48.913625 1475 server_base.cc:1034] running on GCE node
I20260926 11:29:48.914227 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:29:48.915290 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:29:48.916477 1475 hybrid_clock.cc:648] HybridClock initialized: now 1790422188916467 us; error 28 us; skew 500 ppm
Sep 26 11:29:48 dist-test-slave-mrck krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:29:48.919245 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:29:48.920128 1475 webserver.cc:460] Webserver started at http://127.0.98.62:37829/ using document root <none> and password file <none>
I20260926 11:29:48.920640 1475 fs_manager.cc:359] Metadata directory not provided
I20260926 11:29:48.920696 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:29:48.920849 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:29:48.922798 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "5e29c70a9c58451a98b5ccf40f4b2359"
format_stamp: "Formatted at 2026-09-26 11:29:48 on dist-test-slave-mrck"
server_key: "7985717caf96db6cbc65ccf3fcc0c0d8"
server_key_iv: "b73adee11063e17e20e8b5dde936dc98"
server_key_version: "encryptionkey@0"
I20260926 11:29:48.923285 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "5e29c70a9c58451a98b5ccf40f4b2359"
format_stamp: "Formatted at 2026-09-26 11:29:48 on dist-test-slave-mrck"
server_key: "7985717caf96db6cbc65ccf3fcc0c0d8"
server_key_iv: "b73adee11063e17e20e8b5dde936dc98"
server_key_version: "encryptionkey@0"
I20260926 11:29:48.926925 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 11:29:48.929301 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:48.930603 1475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 11:29:48.930701 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "5e29c70a9c58451a98b5ccf40f4b2359"
format_stamp: "Formatted at 2026-09-26 11:29:48 on dist-test-slave-mrck"
server_key: "7985717caf96db6cbc65ccf3fcc0c0d8"
server_key_iv: "b73adee11063e17e20e8b5dde936dc98"
server_key_version: "encryptionkey@0"
I20260926 11:29:48.930787 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:29:48.952682 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:29:48.956077 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:29:48.956233 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:29:48.964223 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44061
I20260926 11:29:48.964236 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44061 every 8 connection(s)
I20260926 11:29:48.965188 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:29:48.967993 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:29:48.968586 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 1475
I20260926 11:29:48.968719 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:29:48.968973 392 external_mini_cluster.cc:1428] Setting key 53af5b5685bcf146964fe6d9d6eaeaf2
I20260926 11:29:48.974318 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e29c70a9c58451a98b5ccf40f4b2359: Bootstrap starting.
I20260926 11:29:48.976986 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e29c70a9c58451a98b5ccf40f4b2359: Neither blocks nor log segments found. Creating new log.
I20260926 11:29:48.977821 1553 log.cc:824] T 00000000000000000000000000000000 P 5e29c70a9c58451a98b5ccf40f4b2359: Log is configured to *not* fsync() on all Append() calls
Sep 26 11:29:48 dist-test-slave-mrck krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:29:48.979638 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e29c70a9c58451a98b5ccf40f4b2359: No bootstrap required, opened a new log
I20260926 11:29:48.982432 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e29c70a9c58451a98b5ccf40f4b2359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e29c70a9c58451a98b5ccf40f4b2359" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44061 } }
I20260926 11:29:48.982766 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e29c70a9c58451a98b5ccf40f4b2359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:29:48.982864 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e29c70a9c58451a98b5ccf40f4b2359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e29c70a9c58451a98b5ccf40f4b2359, State: Initialized, Role: FOLLOWER
I20260926 11:29:48.983377 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e29c70a9c58451a98b5ccf40f4b2359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e29c70a9c58451a98b5ccf40f4b2359" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44061 } }
I20260926 11:29:48.983544 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e29c70a9c58451a98b5ccf40f4b2359 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:29:48.983606 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e29c70a9c58451a98b5ccf40f4b2359 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:29:48.983711 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e29c70a9c58451a98b5ccf40f4b2359 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:29:48.984426 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:48.970354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57818 (local address 127.0.98.62:44061)
0926 11:29:48.970796 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:48.970810 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:48.970844 (+ 34us) server_negotiation.cc:407] Connection header received
0926 11:29:48.971480 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:48.971497 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:48.971896 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:48.972177 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:48.973074 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:48.974058 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:48.975160 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:48.975615 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:48.975807 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:48.978700 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:48.978706 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:48.978719 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:48.978747 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:48.981244 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:48.981775 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:48.981782 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:48.981795 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:48.981920 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:48.982250 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:48.982254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:48.982255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:48.982644 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:48.982879 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:48.983160 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 11:29:48.983442 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":125,"threads_started":1}
I20260926 11:29:48.984799 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e29c70a9c58451a98b5ccf40f4b2359 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e29c70a9c58451a98b5ccf40f4b2359" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44061 } }
I20260926 11:29:48.985155 1553 leader_election.cc:302] T 00000000000000000000000000000000 P 5e29c70a9c58451a98b5ccf40f4b2359 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e29c70a9c58451a98b5ccf40f4b2359; no voters:
I20260926 11:29:48.985466 1553 leader_election.cc:288] T 00000000000000000000000000000000 P 5e29c70a9c58451a98b5ccf40f4b2359 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:29:48.985574 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e29c70a9c58451a98b5ccf40f4b2359 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:29:48.985795 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e29c70a9c58451a98b5ccf40f4b2359 [term 1 LEADER]: Becoming Leader. State: Replica: 5e29c70a9c58451a98b5ccf40f4b2359, State: Running, Role: LEADER
I20260926 11:29:48.986089 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e29c70a9c58451a98b5ccf40f4b2359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e29c70a9c58451a98b5ccf40f4b2359" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44061 } }
I20260926 11:29:48.986856 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e29c70a9c58451a98b5ccf40f4b2359 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:29:48.988153 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e29c70a9c58451a98b5ccf40f4b2359 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e29c70a9c58451a98b5ccf40f4b2359. Latest consensus state: current_term: 1 leader_uuid: "5e29c70a9c58451a98b5ccf40f4b2359" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e29c70a9c58451a98b5ccf40f4b2359" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44061 } } }
I20260926 11:29:48.988132 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e29c70a9c58451a98b5ccf40f4b2359 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e29c70a9c58451a98b5ccf40f4b2359" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e29c70a9c58451a98b5ccf40f4b2359" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44061 } } }
I20260926 11:29:48.988296 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e29c70a9c58451a98b5ccf40f4b2359 [sys.catalog]: This master's current role is: LEADER
I20260926 11:29:48.988296 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e29c70a9c58451a98b5ccf40f4b2359 [sys.catalog]: This master's current role is: LEADER
I20260926 11:29:48.989125 1569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:29:48.991714 1569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:29:48.996973 1569 catalog_manager.cc:1347] Generated new cluster ID: bc4e545a034e411195dba4f96687a81d
I20260926 11:29:48.997067 1569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:29:49.011916 1569 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:29:49.012861 1569 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:29:49.019676 1569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e29c70a9c58451a98b5ccf40f4b2359: Generated new TSK 0
I20260926 11:29:49.020387 1569 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:29:49.050704 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44061
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:29:49.164355 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:29:49.164610 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:29:49.164659 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:29:49.168686 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:29:49.168740 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:29:49.168780 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 11:29:49.168812 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:29:49.173868 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:29:49.174808 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:29:49.175799 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:29:49.185212 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:29:49.185324 1579 server_base.cc:1034] running on GCE node
W20260926 11:29:49.185253 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:29:49.185262 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:29:49.185930 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:29:49.186514 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:29:49.187666 1579 hybrid_clock.cc:648] HybridClock initialized: now 1790422189187653 us; error 26 us; skew 500 ppm
Sep 26 11:29:49 dist-test-slave-mrck krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:29:49.194653 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:29:49.195546 1579 webserver.cc:460] Webserver started at http://127.0.98.1:43591/ using document root <none> and password file <none>
I20260926 11:29:49.196219 1579 fs_manager.cc:359] Metadata directory not provided
I20260926 11:29:49.196308 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:29:49.196550 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:29:49.198559 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-0/data/instance:
uuid: "e10e18126798422f819da4fc40b3a045"
format_stamp: "Formatted at 2026-09-26 11:29:49 on dist-test-slave-mrck"
server_key: "7fcd95653746c95d5b8722126cdec735"
server_key_iv: "763d2d863aacfac653c1594875ba6ddf"
server_key_version: "encryptionkey@0"
I20260926 11:29:49.199117 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e10e18126798422f819da4fc40b3a045"
format_stamp: "Formatted at 2026-09-26 11:29:49 on dist-test-slave-mrck"
server_key: "7fcd95653746c95d5b8722126cdec735"
server_key_iv: "763d2d863aacfac653c1594875ba6ddf"
server_key_version: "encryptionkey@0"
I20260926 11:29:49.202800 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 11:29:49.205386 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:49.206796 1579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 11:29:49.206959 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "e10e18126798422f819da4fc40b3a045"
format_stamp: "Formatted at 2026-09-26 11:29:49 on dist-test-slave-mrck"
server_key: "7fcd95653746c95d5b8722126cdec735"
server_key_iv: "763d2d863aacfac653c1594875ba6ddf"
server_key_version: "encryptionkey@0"
I20260926 11:29:49.207080 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:29:49.233354 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:29:49.236773 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:29:49.236976 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:29:49.238392 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:29:49.238466 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:49.238531 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:29:49.238582 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 11:29:49 dist-test-slave-mrck krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:29:49.249625 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41165
I20260926 11:29:49.249702 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41165 every 8 connection(s)
I20260926 11:29:49.250749 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:29:49.252753 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:49.239301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49267 (local address 127.0.98.62:44061)
0926 11:29:49.239425 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:49.239429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:49.240590 (+ 1161us) server_negotiation.cc:407] Connection header received
0926 11:29:49.241775 (+ 1185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:49.241782 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:49.241852 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:49.241957 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:49.243566 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.244238 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.245492 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.245915 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.246043 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:49.248829 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:49.248833 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:49.248836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:49.248865 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:49.250581 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:49.251098 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:49.251102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:49.251104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:49.251213 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:49.251630 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:49.251633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:49.251635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:49.251887 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:49.251998 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:49.252467 (+ 469us) server_negotiation.cc:299] Negotiation successful
0926 11:29:49.252585 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 11:29:49.253480 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:49.239948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44061 (local address 127.0.98.1:49267)
0926 11:29:49.240438 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:49.240480 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:49.241515 (+ 1035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:49.242163 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:49.242176 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:49.242601 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:29:49.243346 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:49.243367 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.244384 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.244388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:49.245360 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:49.245369 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.246093 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.246099 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:49.246325 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:49.246993 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:29:49.247008 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:29:49.248662 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:29:49.250685 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:49.250689 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:49.250700 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:49.251002 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:49.251325 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:49.251331 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:49.251335 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:49.251517 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:49.251997 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:29:49.252003 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:29:49.252233 (+ 230us) client_negotiation.cc:769] Sending connection context
0926 11:29:49.252414 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 11:29:49.252719 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":144,"threads_started":1}
I20260926 11:29:49.254655 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44061
I20260926 11:29:49.254806 1710 heartbeater.cc:464] Registering TS with master...
I20260926 11:29:49.255280 1710 heartbeater.cc:511] Master 127.0.98.62:44061 requested a full tablet report, sending...
I20260926 11:29:49.256973 1506 ts_manager.cc:194] Registered new tserver with Master: e10e18126798422f819da4fc40b3a045 (127.0.98.1:41165)
I20260926 11:29:49.257120 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 1579
I20260926 11:29:49.257225 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:29:49.257486 392 external_mini_cluster.cc:1428] Setting key 55e7bf4f1d6ce37771ad083846f4ed1f
I20260926 11:29:49.258349 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49267
I20260926 11:29:49.264634 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:49.256858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59685 (local address 127.0.98.62:44061)
0926 11:29:49.256973 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:49.256978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:49.257214 (+ 236us) server_negotiation.cc:407] Connection header received
0926 11:29:49.257357 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:49.257360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:49.257407 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:49.257489 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:49.257976 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.258663 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.259651 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.259924 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.260059 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:49.261316 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:49.261319 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:49.261321 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:49.261341 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:49.262981 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:49.263486 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:49.263489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:49.263490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:49.263556 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:49.263879 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:49.263881 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:49.263882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:49.264035 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:49.264110 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:49.264428 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 11:29:49.264521 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 11:29:49.264809 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:49.256756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44061 (local address 127.0.98.1:59685)
0926 11:29:49.257139 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:49.257153 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:49.257275 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:49.257511 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:49.257514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:49.257727 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:29:49.257846 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:49.257852 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.258817 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.258819 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:49.259540 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:49.259548 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.260063 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.260067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:49.260153 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:49.260705 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:29:49.260724 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:29:49.261188 (+ 464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:29:49.263111 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:49.263116 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:49.263120 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:49.263380 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:49.263645 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:49.263651 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:49.263656 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:49.263784 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:49.264130 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:29:49.264135 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:29:49.264249 (+ 114us) client_negotiation.cc:769] Sending connection context
0926 11:29:49.264381 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 11:29:49.264550 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":111,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:29:49.283416 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44061
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:29:49.410717 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:29:49.411024 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:29:49.411106 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:29:49.415228 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:29:49.415283 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:29:49.415355 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 11:29:49.415427 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:29:49.420504 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:29:49.421520 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:29:49.422533 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:29:49.432045 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:29:49.432054 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:29:49.432052 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:29:49.432318 1719 server_base.cc:1034] running on GCE node
I20260926 11:29:49.432806 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:29:49.433365 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:29:49.434559 1719 hybrid_clock.cc:648] HybridClock initialized: now 1790422189434545 us; error 25 us; skew 500 ppm
Sep 26 11:29:49 dist-test-slave-mrck krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:29:49.437317 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:29:49.438154 1719 webserver.cc:460] Webserver started at http://127.0.98.2:46037/ using document root <none> and password file <none>
I20260926 11:29:49.438720 1719 fs_manager.cc:359] Metadata directory not provided
I20260926 11:29:49.438823 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:29:49.439060 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:29:49.441056 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-1/data/instance:
uuid: "a9dfcdf25b434d37bea03047ca294303"
format_stamp: "Formatted at 2026-09-26 11:29:49 on dist-test-slave-mrck"
server_key: "479244e5b3eafdd18f539f277702a243"
server_key_iv: "a97ef5e6100e4cf08edfd7dd1f01afc7"
server_key_version: "encryptionkey@0"
I20260926 11:29:49.441612 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a9dfcdf25b434d37bea03047ca294303"
format_stamp: "Formatted at 2026-09-26 11:29:49 on dist-test-slave-mrck"
server_key: "479244e5b3eafdd18f539f277702a243"
server_key_iv: "a97ef5e6100e4cf08edfd7dd1f01afc7"
server_key_version: "encryptionkey@0"
I20260926 11:29:49.445235 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:29:49.447674 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:49.449024 1719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:29:49.449146 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "a9dfcdf25b434d37bea03047ca294303"
format_stamp: "Formatted at 2026-09-26 11:29:49 on dist-test-slave-mrck"
server_key: "479244e5b3eafdd18f539f277702a243"
server_key_iv: "a97ef5e6100e4cf08edfd7dd1f01afc7"
server_key_version: "encryptionkey@0"
I20260926 11:29:49.449254 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:29:49.458060 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:29:49.460852 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:29:49.461050 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:29:49.462553 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:29:49.462620 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:49.462685 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:29:49.462718 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 11:29:49 dist-test-slave-mrck krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:29:49.473562 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39351
I20260926 11:29:49.474615 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:29:49.474810 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39351 every 8 connection(s)
I20260926 11:29:49.476212 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:49.463416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46261 (local address 127.0.98.62:44061)
0926 11:29:49.463546 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:49.463550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:49.464567 (+ 1017us) server_negotiation.cc:407] Connection header received
0926 11:29:49.465677 (+ 1110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:49.465681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:49.465737 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:49.465851 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:49.467428 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.468216 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.469536 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.469769 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.469894 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:49.472378 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:49.472381 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:49.472384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:49.472409 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:49.473988 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:49.474597 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:49.474601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:49.474603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:49.474697 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:49.475124 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:49.475127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:49.475130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:49.475338 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:49.475431 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:49.475990 (+ 559us) server_negotiation.cc:299] Negotiation successful
0926 11:29:49.476093 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 11:29:49.476982 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:49.463949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44061 (local address 127.0.98.2:46261)
0926 11:29:49.464397 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:49.464436 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:49.465414 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:49.466077 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:49.466087 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:49.466559 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:29:49.467261 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:49.467282 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.468341 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.468344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:49.469399 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:49.469409 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.469943 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.469947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:49.470111 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:49.470708 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:29:49.470723 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:29:49.472213 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:29:49.474134 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:49.474139 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:49.474154 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:49.474492 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:49.474801 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:49.474807 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:49.474809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:49.475006 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:49.475489 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:29:49.475495 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:29:49.475704 (+ 209us) client_negotiation.cc:769] Sending connection context
0926 11:29:49.475919 (+ 215us) client_negotiation.cc:240] Negotiation successful
0926 11:29:49.476159 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":135,"threads_started":1}
I20260926 11:29:49.478212 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44061
I20260926 11:29:49.478339 1850 heartbeater.cc:464] Registering TS with master...
I20260926 11:29:49.478847 1850 heartbeater.cc:511] Master 127.0.98.62:44061 requested a full tablet report, sending...
I20260926 11:29:49.479489 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 1719
I20260926 11:29:49.479606 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:29:49.479923 392 external_mini_cluster.cc:1428] Setting key 6db86ecf99c0d7fba579b50d5d288869
I20260926 11:29:49.479899 1506 ts_manager.cc:194] Registered new tserver with Master: a9dfcdf25b434d37bea03047ca294303 (127.0.98.2:39351)
I20260926 11:29:49.480530 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46261
I20260926 11:29:49.488157 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:49.480690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44061 (local address 127.0.98.2:50001)
0926 11:29:49.481025 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:49.481041 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:49.481133 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:49.481365 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:49.481368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:49.481561 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:29:49.481650 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:49.481655 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.482372 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.482375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:49.482950 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:49.482957 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.483412 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.483416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:49.483506 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:49.484054 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:29:49.484074 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:29:49.484426 (+ 352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:29:49.486423 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:49.486427 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:49.486428 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:49.486615 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:49.486934 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:49.486937 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:49.486941 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:49.487036 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:49.487590 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:29:49.487592 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:29:49.487662 (+ 70us) client_negotiation.cc:769] Sending connection context
0926 11:29:49.487820 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 11:29:49.487922 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":122,"threads_started":1}
I20260926 11:29:49.488235 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:49.480594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50001 (local address 127.0.98.62:44061)
0926 11:29:49.480698 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:49.480702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:49.481096 (+ 394us) server_negotiation.cc:407] Connection header received
0926 11:29:49.481204 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:49.481208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:49.481247 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:49.481351 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:49.481755 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.482256 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.483064 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.483293 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.483445 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:49.484558 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:49.484565 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:49.484571 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:49.484601 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:49.486270 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:49.486711 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:49.486718 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:49.486721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:49.486814 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:49.487127 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:49.487132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:49.487136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:49.487486 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:49.487585 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:49.487872 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 11:29:49.488028 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:29:49.506134 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44061
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:29:49.614012 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:29:49.614260 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:29:49.614328 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:29:49.618325 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:29:49.618381 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:29:49.618497 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 11:29:49.618548 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:29:49.623385 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:29:49.624476 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:29:49.625494 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:29:49.635265 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:29:49.635294 1859 server_base.cc:1034] running on GCE node
W20260926 11:29:49.635300 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:29:49.635457 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:29:49.635987 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:29:49.636636 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:29:49.637817 1859 hybrid_clock.cc:648] HybridClock initialized: now 1790422189637784 us; error 46 us; skew 500 ppm
Sep 26 11:29:49 dist-test-slave-mrck krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:29:49.644243 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:29:49.645107 1859 webserver.cc:460] Webserver started at http://127.0.98.3:32875/ using document root <none> and password file <none>
I20260926 11:29:49.645610 1859 fs_manager.cc:359] Metadata directory not provided
I20260926 11:29:49.645663 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:29:49.645821 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:29:49.647881 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-2/data/instance:
uuid: "0e01082f3d2642f695a65ec8e1231210"
format_stamp: "Formatted at 2026-09-26 11:29:49 on dist-test-slave-mrck"
server_key: "761be2d06c5f5b2ca5444f8cd7a7f26d"
server_key_iv: "41d23f1c3e795d4a33c084b3559f3d53"
server_key_version: "encryptionkey@0"
I20260926 11:29:49.648437 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0e01082f3d2642f695a65ec8e1231210"
format_stamp: "Formatted at 2026-09-26 11:29:49 on dist-test-slave-mrck"
server_key: "761be2d06c5f5b2ca5444f8cd7a7f26d"
server_key_iv: "41d23f1c3e795d4a33c084b3559f3d53"
server_key_version: "encryptionkey@0"
I20260926 11:29:49.651968 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:29:49.654381 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:49.655720 1859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:29:49.655884 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "0e01082f3d2642f695a65ec8e1231210"
format_stamp: "Formatted at 2026-09-26 11:29:49 on dist-test-slave-mrck"
server_key: "761be2d06c5f5b2ca5444f8cd7a7f26d"
server_key_iv: "41d23f1c3e795d4a33c084b3559f3d53"
server_key_version: "encryptionkey@0"
I20260926 11:29:49.655992 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:29:49.689754 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:29:49.692718 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:29:49.692920 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:29:49.694327 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:29:49.694378 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:49.694418 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:29:49.694473 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 11:29:49 dist-test-slave-mrck krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422189, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:29:49.705160 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39455
I20260926 11:29:49.705176 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39455 every 8 connection(s)
I20260926 11:29:49.706049 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:29:49.708516 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:49.695351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42431 (local address 127.0.98.62:44061)
0926 11:29:49.695497 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:49.695502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:49.696471 (+ 969us) server_negotiation.cc:407] Connection header received
0926 11:29:49.697656 (+ 1185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:49.697659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:49.697715 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:49.697815 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:49.699445 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.700208 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.701507 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.701846 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.701993 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:49.705019 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:49.705022 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:49.705025 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:49.705059 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:49.706641 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:49.707196 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:49.707199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:49.707200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:49.707289 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:49.707536 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:49.707538 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:49.707539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:49.707689 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:49.707815 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:49.708282 (+ 467us) server_negotiation.cc:299] Negotiation successful
0926 11:29:49.708378 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 11:29:49.709290 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:49.695814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44061 (local address 127.0.98.3:42431)
0926 11:29:49.696299 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:49.696348 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:49.697416 (+ 1068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:49.698004 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:49.698015 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:49.698466 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:29:49.699229 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:49.699248 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.700340 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.700344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:49.701380 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:49.701390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.702061 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.702066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:49.702259 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:49.702926 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:29:49.702949 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:29:49.704809 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:29:49.706768 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:49.706773 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:49.706787 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:49.707101 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:49.707368 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:49.707370 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:49.707371 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:49.707482 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:49.707823 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:29:49.707828 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:29:49.708036 (+ 208us) client_negotiation.cc:769] Sending connection context
0926 11:29:49.708228 (+ 192us) client_negotiation.cc:240] Negotiation successful
0926 11:29:49.708529 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":146,"threads_started":1}
I20260926 11:29:49.710573 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44061
I20260926 11:29:49.710775 1990 heartbeater.cc:464] Registering TS with master...
I20260926 11:29:49.711302 1990 heartbeater.cc:511] Master 127.0.98.62:44061 requested a full tablet report, sending...
I20260926 11:29:49.712359 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 1859
I20260926 11:29:49.712378 1506 ts_manager.cc:194] Registered new tserver with Master: 0e01082f3d2642f695a65ec8e1231210 (127.0.98.3:39455)
I20260926 11:29:49.712473 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:29:49.712790 392 external_mini_cluster.cc:1428] Setting key 5c31c8fa467571068f6e65a6fd8dd847
I20260926 11:29:49.713155 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42431
I20260926 11:29:49.714794 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:29:49.720463 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:49.712769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44061 (local address 127.0.98.3:54279)
0926 11:29:49.713094 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:49.713108 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:49.713275 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:49.713538 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:49.713540 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:49.713744 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:29:49.713923 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:49.713929 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.714873 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.714879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:49.715590 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:49.715598 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.716261 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.716264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:49.716332 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:49.716848 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:29:49.716859 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:29:49.717163 (+ 304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:29:49.718945 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:49.718947 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:49.718949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:49.719129 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:49.719461 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:49.719467 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:49.719470 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:49.719551 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:49.719994 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:29:49.719996 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:29:49.720065 (+ 69us) client_negotiation.cc:769] Sending connection context
0926 11:29:49.720145 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 11:29:49.720247 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":20,"thread_start_us":84,"threads_started":1}
I20260926 11:29:49.720569 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:49.712860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54279 (local address 127.0.98.62:44061)
0926 11:29:49.712969 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:49.712973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:49.713171 (+ 198us) server_negotiation.cc:407] Connection header received
0926 11:29:49.713383 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:49.713386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:49.713446 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:49.713522 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:49.714031 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.714743 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.715705 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.716150 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.716278 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:49.717294 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:49.717300 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:49.717304 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:49.717335 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:49.718842 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:49.719231 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:49.719237 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:49.719241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:49.719347 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:49.719648 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:49.719654 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:49.719658 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:49.719895 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:49.719999 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:49.720222 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 11:29:49.720340 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 11:29:49 dist-test-slave-mrck krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422189, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 11:29:49 dist-test-slave-mrck krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422189, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:29:49.749152 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:49.740602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41684 (local address 127.0.98.62:44061)
0926 11:29:49.740799 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:49.740806 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:49.740827 (+ 21us) server_negotiation.cc:407] Connection header received
0926 11:29:49.740885 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:49.740888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:49.740954 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:49.741127 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:49.742027 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.742674 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.743470 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.743669 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.743844 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:49.746288 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:49.746292 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:49.746294 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:49.746316 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:49.747539 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:49.747991 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:49.747994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:49.747995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:49.748071 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:49.748356 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:49.748359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:49.748361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:49.748568 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:49.748678 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:49.748850 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 11:29:49.748946 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 11:29:49.752206 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:29:49.755645 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:29:49.769702 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:49.764118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57030 (local address 127.0.98.2:39351)
0926 11:29:49.764421 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:49.764434 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:49.764884 (+ 450us) server_negotiation.cc:407] Connection header received
0926 11:29:49.765142 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:49.765150 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:49.765368 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:49.765534 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:29:49.765802 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.766816 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.768242 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.769049 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.769150 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:49.769304 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:49.769472 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 11:29:49.769557 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":127,"threads_started":1}
I20260926 11:29:49.769726 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:49.763977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39351 (local address 127.0.0.1:57030)
0926 11:29:49.764741 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:49.764790 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:49.764940 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:49.765515 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:49.765520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:49.765544 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:29:49.765684 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:49.765691 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.766915 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.766921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:49.768104 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:49.768114 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.769156 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.769160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:49.769276 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:49.769316 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 11:29:49.769442 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 11:29:49.769549 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":22,"thread_start_us":69,"threads_started":1}
I20260926 11:29:49.770299 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:49.763977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39455 (local address 127.0.0.1:37484)
0926 11:29:49.764748 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:49.764790 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:49.764917 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:49.766097 (+ 1180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:49.766100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:49.766117 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:29:49.766214 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:49.766223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.767554 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.767558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:49.768704 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:49.768714 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.769935 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.769939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:49.770036 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:49.770061 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 11:29:49.770125 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 11:29:49.770184 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":567,"thread_start_us":96,"threads_started":1}
I20260926 11:29:49.770610 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:49.764865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41165 (local address 127.0.0.1:39920)
0926 11:29:49.765430 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:49.765443 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:49.765538 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:49.766373 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:49.766376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:49.766392 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:29:49.766474 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:49.766480 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.767823 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.767826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:49.768929 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:49.768939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.770235 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.770238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:49.770295 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:49.770311 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 11:29:49.770395 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 11:29:49.770470 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":25,"thread_start_us":92,"threads_started":1}
I20260926 11:29:49.770730 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:49.764179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37484 (local address 127.0.98.3:39455)
0926 11:29:49.765758 (+ 1579us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:49.765765 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:49.765783 (+ 18us) server_negotiation.cc:407] Connection header received
0926 11:29:49.765837 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:49.765840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:49.765972 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:49.766122 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:29:49.766334 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.767439 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.768824 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.769601 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.769693 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:49.769866 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:49.770500 (+ 634us) server_negotiation.cc:299] Negotiation successful
0926 11:29:49.770588 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1498,"thread_start_us":111,"threads_started":1}
I20260926 11:29:49.770980 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:49.765012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39920 (local address 127.0.98.1:41165)
0926 11:29:49.765402 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:49.765408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:49.765824 (+ 416us) server_negotiation.cc:407] Connection header received
0926 11:29:49.765902 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:49.765908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:49.766116 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:49.766253 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:29:49.766593 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.767672 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.769050 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:49.769966 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:49.770059 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:49.770174 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:49.770776 (+ 602us) server_negotiation.cc:299] Negotiation successful
0926 11:29:49.770868 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":89,"threads_started":1}
I20260926 11:29:49.772763 1784 tablet_service.cc:1467] Processing CreateTablet for tablet f439957feac5449899622b35072f7465 (DEFAULT_TABLE table=test-table [id=a997a9f371f8417bba713ba194db0c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:29:49.773617 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f439957feac5449899622b35072f7465. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:29:49.773800 1924 tablet_service.cc:1467] Processing CreateTablet for tablet f439957feac5449899622b35072f7465 (DEFAULT_TABLE table=test-table [id=a997a9f371f8417bba713ba194db0c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:29:49.773797 1640 tablet_service.cc:1467] Processing CreateTablet for tablet f439957feac5449899622b35072f7465 (DEFAULT_TABLE table=test-table [id=a997a9f371f8417bba713ba194db0c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:29:49.774724 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f439957feac5449899622b35072f7465. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:29:49.774719 1640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f439957feac5449899622b35072f7465. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:29:49.780580 2013 tablet_bootstrap.cc:492] T f439957feac5449899622b35072f7465 P a9dfcdf25b434d37bea03047ca294303: Bootstrap starting.
I20260926 11:29:49.781800 2015 tablet_bootstrap.cc:492] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210: Bootstrap starting.
I20260926 11:29:49.782341 2014 tablet_bootstrap.cc:492] T f439957feac5449899622b35072f7465 P e10e18126798422f819da4fc40b3a045: Bootstrap starting.
I20260926 11:29:49.783247 2013 tablet_bootstrap.cc:654] T f439957feac5449899622b35072f7465 P a9dfcdf25b434d37bea03047ca294303: Neither blocks nor log segments found. Creating new log.
I20260926 11:29:49.783895 2015 tablet_bootstrap.cc:654] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210: Neither blocks nor log segments found. Creating new log.
I20260926 11:29:49.784111 2014 tablet_bootstrap.cc:654] T f439957feac5449899622b35072f7465 P e10e18126798422f819da4fc40b3a045: Neither blocks nor log segments found. Creating new log.
I20260926 11:29:49.784219 2013 log.cc:824] T f439957feac5449899622b35072f7465 P a9dfcdf25b434d37bea03047ca294303: Log is configured to *not* fsync() on all Append() calls
I20260926 11:29:49.784688 2015 log.cc:824] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210: Log is configured to *not* fsync() on all Append() calls
I20260926 11:29:49.784850 2014 log.cc:824] T f439957feac5449899622b35072f7465 P e10e18126798422f819da4fc40b3a045: Log is configured to *not* fsync() on all Append() calls
I20260926 11:29:49.786377 2013 tablet_bootstrap.cc:492] T f439957feac5449899622b35072f7465 P a9dfcdf25b434d37bea03047ca294303: No bootstrap required, opened a new log
I20260926 11:29:49.786571 2013 ts_tablet_manager.cc:1397] T f439957feac5449899622b35072f7465 P a9dfcdf25b434d37bea03047ca294303: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 11:29:49.786690 2014 tablet_bootstrap.cc:492] T f439957feac5449899622b35072f7465 P e10e18126798422f819da4fc40b3a045: No bootstrap required, opened a new log
I20260926 11:29:49.786855 2014 ts_tablet_manager.cc:1397] T f439957feac5449899622b35072f7465 P e10e18126798422f819da4fc40b3a045: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 11:29:49.790549 2013 raft_consensus.cc:348] T f439957feac5449899622b35072f7465 P a9dfcdf25b434d37bea03047ca294303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } } peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } }
I20260926 11:29:49.790819 2013 raft_consensus.cc:374] T f439957feac5449899622b35072f7465 P a9dfcdf25b434d37bea03047ca294303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:29:49.790867 2013 raft_consensus.cc:729] T f439957feac5449899622b35072f7465 P a9dfcdf25b434d37bea03047ca294303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9dfcdf25b434d37bea03047ca294303, State: Initialized, Role: FOLLOWER
I20260926 11:29:49.791006 2014 raft_consensus.cc:348] T f439957feac5449899622b35072f7465 P e10e18126798422f819da4fc40b3a045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } } peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } }
I20260926 11:29:49.791231 2014 raft_consensus.cc:374] T f439957feac5449899622b35072f7465 P e10e18126798422f819da4fc40b3a045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:29:49.791298 2014 raft_consensus.cc:729] T f439957feac5449899622b35072f7465 P e10e18126798422f819da4fc40b3a045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e10e18126798422f819da4fc40b3a045, State: Initialized, Role: FOLLOWER
I20260926 11:29:49.791433 2013 consensus_queue.cc:260] T f439957feac5449899622b35072f7465 P a9dfcdf25b434d37bea03047ca294303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } } peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } }
I20260926 11:29:49.791708 2014 consensus_queue.cc:260] T f439957feac5449899622b35072f7465 P e10e18126798422f819da4fc40b3a045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } } peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } }
I20260926 11:29:49.792337 2015 tablet_bootstrap.cc:492] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210: No bootstrap required, opened a new log
I20260926 11:29:49.792444 2013 ts_tablet_manager.cc:1428] T f439957feac5449899622b35072f7465 P a9dfcdf25b434d37bea03047ca294303: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260926 11:29:49.792444 2014 ts_tablet_manager.cc:1428] T f439957feac5449899622b35072f7465 P e10e18126798422f819da4fc40b3a045: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260926 11:29:49.792559 2015 ts_tablet_manager.cc:1397] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.003s
I20260926 11:29:49.792796 1850 heartbeater.cc:503] Master 127.0.98.62:44061 was elected leader, sending a full tablet report...
I20260926 11:29:49.792809 1710 heartbeater.cc:503] Master 127.0.98.62:44061 was elected leader, sending a full tablet report...
I20260926 11:29:49.795938 2015 raft_consensus.cc:348] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } } peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } }
I20260926 11:29:49.796205 2015 raft_consensus.cc:374] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:29:49.796263 2015 raft_consensus.cc:729] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e01082f3d2642f695a65ec8e1231210, State: Initialized, Role: FOLLOWER
I20260926 11:29:49.796738 2015 consensus_queue.cc:260] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } } peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } }
I20260926 11:29:49.797343 1990 heartbeater.cc:503] Master 127.0.98.62:44061 was elected leader, sending a full tablet report...
I20260926 11:29:49.797659 2015 ts_tablet_manager.cc:1428] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260926 11:29:49.957268 1991 tablet.cc:2369] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 11:29:49.976033 1851 tablet.cc:2369] T f439957feac5449899622b35072f7465 P a9dfcdf25b434d37bea03047ca294303: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 11:29:50.002605 1711 tablet.cc:2369] T f439957feac5449899622b35072f7465 P e10e18126798422f819da4fc40b3a045: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:29:50.082702 2021 raft_consensus.cc:482] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:29:50.082983 2021 raft_consensus.cc:504] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } } peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } }
I20260926 11:29:50.084177 2021 leader_election.cc:288] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9dfcdf25b434d37bea03047ca294303 (127.0.98.2:39351), e10e18126798422f819da4fc40b3a045 (127.0.98.1:41165)
I20260926 11:29:50.088613 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:50.084520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36505 (local address 127.0.98.2:39351)
0926 11:29:50.084668 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:50.084672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:50.084760 (+ 88us) server_negotiation.cc:407] Connection header received
0926 11:29:50.084938 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:50.084941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:50.084999 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:50.085081 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:29:50.085367 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.085918 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.087329 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.088023 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.088145 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:50.088210 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:50.088391 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 11:29:50.088486 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 11:29:50.088625 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:50.084436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39351 (local address 127.0.98.3:36505)
0926 11:29:50.084671 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:50.084688 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:50.084794 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:50.085095 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:50.085097 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:50.085108 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:29:50.085261 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:50.085268 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.086012 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.086014 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:50.087150 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:50.087160 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.088150 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.088153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:50.088238 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:50.088259 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 11:29:50.088330 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 11:29:50.088436 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":1}
I20260926 11:29:50.089049 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:50.084523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43993 (local address 127.0.98.1:41165)
0926 11:29:50.084647 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:50.084651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:50.084923 (+ 272us) server_negotiation.cc:407] Connection header received
0926 11:29:50.085012 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:50.085016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:50.085073 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:50.085138 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:29:50.085735 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.086300 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.087681 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.088373 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.088499 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:50.088556 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:50.088800 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 11:29:50.088896 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 11:29:50.089287 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f439957feac5449899622b35072f7465" candidate_uuid: "0e01082f3d2642f695a65ec8e1231210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9dfcdf25b434d37bea03047ca294303" is_pre_election: true
I20260926 11:29:50.089406 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:50.084436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41165 (local address 127.0.98.3:43993)
0926 11:29:50.084824 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:50.084839 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:50.084943 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:50.085152 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:50.085154 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:50.085164 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:29:50.085625 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:50.085631 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.086437 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.086440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:50.087565 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:50.087573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.088539 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.088543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:50.088634 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:50.088666 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 11:29:50.089193 (+ 527us) client_negotiation.cc:240] Negotiation successful
0926 11:29:50.089265 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":66,"thread_start_us":119,"threads_started":1}
I20260926 11:29:50.089629 1804 raft_consensus.cc:2454] T f439957feac5449899622b35072f7465 P a9dfcdf25b434d37bea03047ca294303 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e01082f3d2642f695a65ec8e1231210 in term 0.
I20260926 11:29:50.090049 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f439957feac5449899622b35072f7465" candidate_uuid: "0e01082f3d2642f695a65ec8e1231210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e10e18126798422f819da4fc40b3a045" is_pre_election: true
I20260926 11:29:50.090129 1875 leader_election.cc:302] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e01082f3d2642f695a65ec8e1231210, a9dfcdf25b434d37bea03047ca294303; no voters:
I20260926 11:29:50.090358 1664 raft_consensus.cc:2454] T f439957feac5449899622b35072f7465 P e10e18126798422f819da4fc40b3a045 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e01082f3d2642f695a65ec8e1231210 in term 0.
I20260926 11:29:50.090406 2021 raft_consensus.cc:2781] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:29:50.090544 2021 raft_consensus.cc:482] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:29:50.090602 2021 raft_consensus.cc:3037] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:29:50.091648 2021 raft_consensus.cc:504] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } } peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } }
I20260926 11:29:50.092070 2021 leader_election.cc:288] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210 [CANDIDATE]: Term 1 election: Requested vote from peers a9dfcdf25b434d37bea03047ca294303 (127.0.98.2:39351), e10e18126798422f819da4fc40b3a045 (127.0.98.1:41165)
I20260926 11:29:50.092384 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f439957feac5449899622b35072f7465" candidate_uuid: "0e01082f3d2642f695a65ec8e1231210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e10e18126798422f819da4fc40b3a045"
I20260926 11:29:50.092413 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f439957feac5449899622b35072f7465" candidate_uuid: "0e01082f3d2642f695a65ec8e1231210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9dfcdf25b434d37bea03047ca294303"
I20260926 11:29:50.092519 1804 raft_consensus.cc:3037] T f439957feac5449899622b35072f7465 P a9dfcdf25b434d37bea03047ca294303 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:29:50.092518 1664 raft_consensus.cc:3037] T f439957feac5449899622b35072f7465 P e10e18126798422f819da4fc40b3a045 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:29:50.093773 1664 raft_consensus.cc:2454] T f439957feac5449899622b35072f7465 P e10e18126798422f819da4fc40b3a045 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e01082f3d2642f695a65ec8e1231210 in term 1.
I20260926 11:29:50.093773 1804 raft_consensus.cc:2454] T f439957feac5449899622b35072f7465 P a9dfcdf25b434d37bea03047ca294303 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e01082f3d2642f695a65ec8e1231210 in term 1.
I20260926 11:29:50.094110 1877 leader_election.cc:302] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e01082f3d2642f695a65ec8e1231210, e10e18126798422f819da4fc40b3a045; no voters:
I20260926 11:29:50.094331 2021 raft_consensus.cc:2781] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:29:50.094539 2021 raft_consensus.cc:686] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210 [term 1 LEADER]: Becoming Leader. State: Replica: 0e01082f3d2642f695a65ec8e1231210, State: Running, Role: LEADER
I20260926 11:29:50.094856 2021 consensus_queue.cc:237] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } } peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } }
I20260926 11:29:50.097950 1507 catalog_manager.cc:5495] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e01082f3d2642f695a65ec8e1231210 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0e01082f3d2642f695a65ec8e1231210" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } health_report { overall_health: UNKNOWN } } }
I20260926 11:29:50.143282 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:50.139568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37496 (local address 127.0.98.3:39455)
0926 11:29:50.139763 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:50.139767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:50.139834 (+ 67us) server_negotiation.cc:407] Connection header received
0926 11:29:50.140101 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:50.140103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:50.140158 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:50.140247 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:29:50.140508 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.140997 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.141886 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.142110 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.142187 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:50.142466 (+ 279us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:29:50.142875 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:29:50.142973 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:50.143062 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 11:29:50.143107 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 11:29:50.148701 1664 raft_consensus.cc:1261] T f439957feac5449899622b35072f7465 P e10e18126798422f819da4fc40b3a045 [term 1 FOLLOWER]: Refusing update from remote peer 0e01082f3d2642f695a65ec8e1231210: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:29:50.148702 1804 raft_consensus.cc:1261] T f439957feac5449899622b35072f7465 P a9dfcdf25b434d37bea03047ca294303 [term 1 FOLLOWER]: Refusing update from remote peer 0e01082f3d2642f695a65ec8e1231210: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:29:50.149384 2023 consensus_queue.cc:1035] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:29:50.149564 2021 consensus_queue.cc:1035] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:29:50.157080 2027 mvcc.cc:205] Tried to move back new op lower bound from 7333569290842431488 to 7333569290630823936. Current Snapshot: MvccSnapshot[applied={T|T < 7333569290842431488}]
I20260926 11:29:50.157521 2026 mvcc.cc:205] Tried to move back new op lower bound from 7333569290842431488 to 7333569290630823936. Current Snapshot: MvccSnapshot[applied={T|T < 7333569290842431488}]
I20260926 11:29:50.167701 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:29:50.167902 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:29:50.170148 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e29c70a9c58451a98b5ccf40f4b2359: Sending DeleteTablet for 3 replicas of tablet f439957feac5449899622b35072f7465
I20260926 11:29:50.170861 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet f439957feac5449899622b35072f7465 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:29:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57030
I20260926 11:29:50.170886 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet f439957feac5449899622b35072f7465 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:29:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37484
I20260926 11:29:50.170985 1640 tablet_service.cc:1514] Processing DeleteTablet for tablet f439957feac5449899622b35072f7465 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:29:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39920
I20260926 11:29:50.171396 2038 tablet_replica.cc:330] T f439957feac5449899622b35072f7465 P e10e18126798422f819da4fc40b3a045: stopping tablet replica
I20260926 11:29:50.171607 2037 tablet_replica.cc:330] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210: stopping tablet replica
I20260926 11:29:50.171713 2038 raft_consensus.cc:2229] T f439957feac5449899622b35072f7465 P e10e18126798422f819da4fc40b3a045 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:29:50.171763 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 11:29:50.171862 2037 raft_consensus.cc:2229] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:29:50.172019 2038 raft_consensus.cc:2258] T f439957feac5449899622b35072f7465 P e10e18126798422f819da4fc40b3a045 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 11:29:50.172137 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:29:50.172211 2037 raft_consensus.cc:2258] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:29:50.173444 2037 ts_tablet_manager.cc:1905] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:29:50.173444 2038 ts_tablet_manager.cc:1905] T f439957feac5449899622b35072f7465 P e10e18126798422f819da4fc40b3a045: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:29:50.176231 2037 ts_tablet_manager.cc:1918] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:29:50.176326 2037 log.cc:1196] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-2/wal/wals/f439957feac5449899622b35072f7465
I20260926 11:29:50.176640 2037 ts_tablet_manager.cc:1939] T f439957feac5449899622b35072f7465 P 0e01082f3d2642f695a65ec8e1231210: Deleting consensus metadata
I20260926 11:29:50.177070 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 8e515f325f8a466f9378deaa25f9d975 (DEFAULT_TABLE table=test-table [id=f321374c77494c039f58f4898ebbd72d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:29:50.177330 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e515f325f8a466f9378deaa25f9d975. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:29:50.177762 2036 tablet_replica.cc:330] T f439957feac5449899622b35072f7465 P a9dfcdf25b434d37bea03047ca294303: stopping tablet replica
I20260926 11:29:50.177870 2038 ts_tablet_manager.cc:1918] T f439957feac5449899622b35072f7465 P e10e18126798422f819da4fc40b3a045: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:29:50.177943 2038 log.cc:1196] T f439957feac5449899622b35072f7465 P e10e18126798422f819da4fc40b3a045: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-0/wal/wals/f439957feac5449899622b35072f7465
I20260926 11:29:50.178103 2036 raft_consensus.cc:2229] T f439957feac5449899622b35072f7465 P a9dfcdf25b434d37bea03047ca294303 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:29:50.178232 2038 ts_tablet_manager.cc:1939] T f439957feac5449899622b35072f7465 P e10e18126798422f819da4fc40b3a045: Deleting consensus metadata
I20260926 11:29:50.178390 2036 raft_consensus.cc:2258] T f439957feac5449899622b35072f7465 P a9dfcdf25b434d37bea03047ca294303 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:29:50.179346 1493 catalog_manager.cc:4841] TS 0e01082f3d2642f695a65ec8e1231210 (127.0.98.3:39455): tablet f439957feac5449899622b35072f7465 (table test-table [id=a997a9f371f8417bba713ba194db0c15]) successfully deleted
I20260926 11:29:50.179718 2013 tablet_bootstrap.cc:492] T 8e515f325f8a466f9378deaa25f9d975 P a9dfcdf25b434d37bea03047ca294303: Bootstrap starting.
I20260926 11:29:50.179994 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 8e515f325f8a466f9378deaa25f9d975 (DEFAULT_TABLE table=test-table [id=f321374c77494c039f58f4898ebbd72d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:29:50.180215 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e515f325f8a466f9378deaa25f9d975. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:29:50.180284 1640 tablet_service.cc:1467] Processing CreateTablet for tablet 8e515f325f8a466f9378deaa25f9d975 (DEFAULT_TABLE table=test-table [id=f321374c77494c039f58f4898ebbd72d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:29:50.180503 1640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e515f325f8a466f9378deaa25f9d975. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:29:50.180495 1493 catalog_manager.cc:4841] TS e10e18126798422f819da4fc40b3a045 (127.0.98.1:41165): tablet f439957feac5449899622b35072f7465 (table test-table [id=a997a9f371f8417bba713ba194db0c15]) successfully deleted
I20260926 11:29:50.180691 2013 tablet_bootstrap.cc:654] T 8e515f325f8a466f9378deaa25f9d975 P a9dfcdf25b434d37bea03047ca294303: Neither blocks nor log segments found. Creating new log.
I20260926 11:29:50.181620 2036 ts_tablet_manager.cc:1905] T f439957feac5449899622b35072f7465 P a9dfcdf25b434d37bea03047ca294303: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:29:50.182313 2015 tablet_bootstrap.cc:492] T 8e515f325f8a466f9378deaa25f9d975 P 0e01082f3d2642f695a65ec8e1231210: Bootstrap starting.
I20260926 11:29:50.182588 2014 tablet_bootstrap.cc:492] T 8e515f325f8a466f9378deaa25f9d975 P e10e18126798422f819da4fc40b3a045: Bootstrap starting.
I20260926 11:29:50.183219 2015 tablet_bootstrap.cc:654] T 8e515f325f8a466f9378deaa25f9d975 P 0e01082f3d2642f695a65ec8e1231210: Neither blocks nor log segments found. Creating new log.
I20260926 11:29:50.183530 2014 tablet_bootstrap.cc:654] T 8e515f325f8a466f9378deaa25f9d975 P e10e18126798422f819da4fc40b3a045: Neither blocks nor log segments found. Creating new log.
I20260926 11:29:50.184226 2013 tablet_bootstrap.cc:492] T 8e515f325f8a466f9378deaa25f9d975 P a9dfcdf25b434d37bea03047ca294303: No bootstrap required, opened a new log
I20260926 11:29:50.184305 2013 ts_tablet_manager.cc:1397] T 8e515f325f8a466f9378deaa25f9d975 P a9dfcdf25b434d37bea03047ca294303: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260926 11:29:50.185000 2014 tablet_bootstrap.cc:492] T 8e515f325f8a466f9378deaa25f9d975 P e10e18126798422f819da4fc40b3a045: No bootstrap required, opened a new log
I20260926 11:29:50.185067 2014 ts_tablet_manager.cc:1397] T 8e515f325f8a466f9378deaa25f9d975 P e10e18126798422f819da4fc40b3a045: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 11:29:50.185155 2013 raft_consensus.cc:348] T 8e515f325f8a466f9378deaa25f9d975 P a9dfcdf25b434d37bea03047ca294303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } } peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } }
I20260926 11:29:50.185256 2013 raft_consensus.cc:374] T 8e515f325f8a466f9378deaa25f9d975 P a9dfcdf25b434d37bea03047ca294303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:29:50.185285 2013 raft_consensus.cc:729] T 8e515f325f8a466f9378deaa25f9d975 P a9dfcdf25b434d37bea03047ca294303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9dfcdf25b434d37bea03047ca294303, State: Initialized, Role: FOLLOWER
I20260926 11:29:50.185859 2015 tablet_bootstrap.cc:492] T 8e515f325f8a466f9378deaa25f9d975 P 0e01082f3d2642f695a65ec8e1231210: No bootstrap required, opened a new log
I20260926 11:29:50.185813 2014 raft_consensus.cc:348] T 8e515f325f8a466f9378deaa25f9d975 P e10e18126798422f819da4fc40b3a045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } } peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } }
I20260926 11:29:50.185932 2015 ts_tablet_manager.cc:1397] T 8e515f325f8a466f9378deaa25f9d975 P 0e01082f3d2642f695a65ec8e1231210: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260926 11:29:50.185933 2014 raft_consensus.cc:374] T 8e515f325f8a466f9378deaa25f9d975 P e10e18126798422f819da4fc40b3a045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:29:50.185979 2014 raft_consensus.cc:729] T 8e515f325f8a466f9378deaa25f9d975 P e10e18126798422f819da4fc40b3a045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e10e18126798422f819da4fc40b3a045, State: Initialized, Role: FOLLOWER
I20260926 11:29:50.186100 2014 consensus_queue.cc:260] T 8e515f325f8a466f9378deaa25f9d975 P e10e18126798422f819da4fc40b3a045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } } peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } }
I20260926 11:29:50.186264 2036 ts_tablet_manager.cc:1918] T f439957feac5449899622b35072f7465 P a9dfcdf25b434d37bea03047ca294303: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:29:50.186339 2036 log.cc:1196] T f439957feac5449899622b35072f7465 P a9dfcdf25b434d37bea03047ca294303: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-1/wal/wals/f439957feac5449899622b35072f7465
I20260926 11:29:50.186311 2015 raft_consensus.cc:348] T 8e515f325f8a466f9378deaa25f9d975 P 0e01082f3d2642f695a65ec8e1231210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } } peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } }
I20260926 11:29:50.186403 2015 raft_consensus.cc:374] T 8e515f325f8a466f9378deaa25f9d975 P 0e01082f3d2642f695a65ec8e1231210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:29:50.186405 2014 ts_tablet_manager.cc:1428] T 8e515f325f8a466f9378deaa25f9d975 P e10e18126798422f819da4fc40b3a045: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 11:29:50.186432 2015 raft_consensus.cc:729] T 8e515f325f8a466f9378deaa25f9d975 P 0e01082f3d2642f695a65ec8e1231210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e01082f3d2642f695a65ec8e1231210, State: Initialized, Role: FOLLOWER
I20260926 11:29:50.186553 2015 consensus_queue.cc:260] T 8e515f325f8a466f9378deaa25f9d975 P 0e01082f3d2642f695a65ec8e1231210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } } peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } }
I20260926 11:29:50.186643 2036 ts_tablet_manager.cc:1939] T f439957feac5449899622b35072f7465 P a9dfcdf25b434d37bea03047ca294303: Deleting consensus metadata
I20260926 11:29:50.186780 2015 ts_tablet_manager.cc:1428] T 8e515f325f8a466f9378deaa25f9d975 P 0e01082f3d2642f695a65ec8e1231210: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 11:29:50.187860 1491 catalog_manager.cc:4841] TS a9dfcdf25b434d37bea03047ca294303 (127.0.98.2:39351): tablet f439957feac5449899622b35072f7465 (table test-table [id=a997a9f371f8417bba713ba194db0c15]) successfully deleted
I20260926 11:29:50.188237 2013 consensus_queue.cc:260] T 8e515f325f8a466f9378deaa25f9d975 P a9dfcdf25b434d37bea03047ca294303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } } peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } }
I20260926 11:29:50.188524 2013 ts_tablet_manager.cc:1428] T 8e515f325f8a466f9378deaa25f9d975 P a9dfcdf25b434d37bea03047ca294303: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260926 11:29:50.279999 2019 raft_consensus.cc:482] T 8e515f325f8a466f9378deaa25f9d975 P a9dfcdf25b434d37bea03047ca294303 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:29:50.280157 2019 raft_consensus.cc:504] T 8e515f325f8a466f9378deaa25f9d975 P a9dfcdf25b434d37bea03047ca294303 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } } peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } }
I20260926 11:29:50.281147 2019 leader_election.cc:288] T 8e515f325f8a466f9378deaa25f9d975 P a9dfcdf25b434d37bea03047ca294303 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e10e18126798422f819da4fc40b3a045 (127.0.98.1:41165), 0e01082f3d2642f695a65ec8e1231210 (127.0.98.3:39455)
I20260926 11:29:50.285820 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:50.281529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55851 (local address 127.0.98.1:41165)
0926 11:29:50.281676 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:50.281682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:50.281794 (+ 112us) server_negotiation.cc:407] Connection header received
0926 11:29:50.282012 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:50.282015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:50.282068 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:50.282131 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:29:50.282558 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.283140 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.284674 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.285243 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.285360 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:50.285426 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:50.285641 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 11:29:50.285710 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 11:29:50.285969 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:50.281934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41305 (local address 127.0.98.3:39455)
0926 11:29:50.282053 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:50.282058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:50.282334 (+ 276us) server_negotiation.cc:407] Connection header received
0926 11:29:50.282400 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:50.282403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:50.282450 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:50.282540 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:29:50.282826 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.283522 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.284915 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.285289 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.285421 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:50.285482 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:50.285778 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 11:29:50.285848 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 11:29:50.285861 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:50.281416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41165 (local address 127.0.98.2:55851)
0926 11:29:50.281674 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:50.281690 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:50.281878 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:50.282282 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:50.282287 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:50.282305 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:29:50.282448 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:50.282453 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.283243 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.283249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:50.284529 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:50.284538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.285389 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.285395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:50.285496 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:50.285526 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 11:29:50.285623 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 11:29:50.285689 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":11,"thread_start_us":97,"threads_started":1}
I20260926 11:29:50.286336 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:50.281792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39455 (local address 127.0.98.2:41305)
0926 11:29:50.282218 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:50.282233 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:50.282312 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:50.282583 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:50.282586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:50.282604 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:29:50.282704 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:50.282709 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.283658 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.283661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:50.284803 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:50.284810 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.285518 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.285522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:50.285626 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:50.285650 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 11:29:50.286112 (+ 462us) client_negotiation.cc:240] Negotiation successful
0926 11:29:50.286183 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":99,"threads_started":1}
I20260926 11:29:50.286571 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e515f325f8a466f9378deaa25f9d975" candidate_uuid: "a9dfcdf25b434d37bea03047ca294303" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e10e18126798422f819da4fc40b3a045" is_pre_election: true
I20260926 11:29:50.286734 1664 raft_consensus.cc:2454] T 8e515f325f8a466f9378deaa25f9d975 P e10e18126798422f819da4fc40b3a045 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9dfcdf25b434d37bea03047ca294303 in term 0.
I20260926 11:29:50.287075 1737 leader_election.cc:302] T 8e515f325f8a466f9378deaa25f9d975 P a9dfcdf25b434d37bea03047ca294303 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9dfcdf25b434d37bea03047ca294303, e10e18126798422f819da4fc40b3a045; no voters:
I20260926 11:29:50.287114 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e515f325f8a466f9378deaa25f9d975" candidate_uuid: "a9dfcdf25b434d37bea03047ca294303" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e01082f3d2642f695a65ec8e1231210" is_pre_election: true
I20260926 11:29:50.287361 2019 raft_consensus.cc:2781] T 8e515f325f8a466f9378deaa25f9d975 P a9dfcdf25b434d37bea03047ca294303 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:29:50.287364 1944 raft_consensus.cc:2454] T 8e515f325f8a466f9378deaa25f9d975 P 0e01082f3d2642f695a65ec8e1231210 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9dfcdf25b434d37bea03047ca294303 in term 0.
I20260926 11:29:50.287427 2019 raft_consensus.cc:482] T 8e515f325f8a466f9378deaa25f9d975 P a9dfcdf25b434d37bea03047ca294303 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:29:50.287487 2019 raft_consensus.cc:3037] T 8e515f325f8a466f9378deaa25f9d975 P a9dfcdf25b434d37bea03047ca294303 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:29:50.288829 2019 raft_consensus.cc:504] T 8e515f325f8a466f9378deaa25f9d975 P a9dfcdf25b434d37bea03047ca294303 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } } peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } }
I20260926 11:29:50.289179 2019 leader_election.cc:288] T 8e515f325f8a466f9378deaa25f9d975 P a9dfcdf25b434d37bea03047ca294303 [CANDIDATE]: Term 1 election: Requested vote from peers e10e18126798422f819da4fc40b3a045 (127.0.98.1:41165), 0e01082f3d2642f695a65ec8e1231210 (127.0.98.3:39455)
I20260926 11:29:50.289707 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e515f325f8a466f9378deaa25f9d975" candidate_uuid: "a9dfcdf25b434d37bea03047ca294303" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e10e18126798422f819da4fc40b3a045"
I20260926 11:29:50.289695 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e515f325f8a466f9378deaa25f9d975" candidate_uuid: "a9dfcdf25b434d37bea03047ca294303" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e01082f3d2642f695a65ec8e1231210"
I20260926 11:29:50.289811 1944 raft_consensus.cc:3037] T 8e515f325f8a466f9378deaa25f9d975 P 0e01082f3d2642f695a65ec8e1231210 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:29:50.289811 1664 raft_consensus.cc:3037] T 8e515f325f8a466f9378deaa25f9d975 P e10e18126798422f819da4fc40b3a045 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:29:50.290855 1664 raft_consensus.cc:2454] T 8e515f325f8a466f9378deaa25f9d975 P e10e18126798422f819da4fc40b3a045 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9dfcdf25b434d37bea03047ca294303 in term 1.
I20260926 11:29:50.290855 1944 raft_consensus.cc:2454] T 8e515f325f8a466f9378deaa25f9d975 P 0e01082f3d2642f695a65ec8e1231210 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9dfcdf25b434d37bea03047ca294303 in term 1.
I20260926 11:29:50.291105 1737 leader_election.cc:302] T 8e515f325f8a466f9378deaa25f9d975 P a9dfcdf25b434d37bea03047ca294303 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9dfcdf25b434d37bea03047ca294303, e10e18126798422f819da4fc40b3a045; no voters:
I20260926 11:29:50.291301 2019 raft_consensus.cc:2781] T 8e515f325f8a466f9378deaa25f9d975 P a9dfcdf25b434d37bea03047ca294303 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:29:50.291539 2019 raft_consensus.cc:686] T 8e515f325f8a466f9378deaa25f9d975 P a9dfcdf25b434d37bea03047ca294303 [term 1 LEADER]: Becoming Leader. State: Replica: a9dfcdf25b434d37bea03047ca294303, State: Running, Role: LEADER
I20260926 11:29:50.291818 2019 consensus_queue.cc:237] T 8e515f325f8a466f9378deaa25f9d975 P a9dfcdf25b434d37bea03047ca294303 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } } peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } }
I20260926 11:29:50.294117 1505 catalog_manager.cc:5495] T 8e515f325f8a466f9378deaa25f9d975 P a9dfcdf25b434d37bea03047ca294303 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9dfcdf25b434d37bea03047ca294303 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a9dfcdf25b434d37bea03047ca294303" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } health_report { overall_health: UNKNOWN } } }
Sep 26 11:29:50 dist-test-slave-mrck krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:29:50.345517 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:50.337542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41690 (local address 127.0.98.62:44061)
0926 11:29:50.337837 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:50.337844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:50.337858 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:29:50.337969 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:50.337971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:50.338011 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:50.338089 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:50.338733 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.339256 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.340064 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.340271 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.340347 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:50.342503 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:50.342506 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:50.342508 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:50.342527 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:50.343837 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:50.344246 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:50.344248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:50.344249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:50.344326 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:50.344633 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:50.344636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:50.344637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:50.344830 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:50.344948 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:50.345225 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 11:29:50.345342 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":142,"threads_started":1}
I20260926 11:29:50.345521 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:50.337463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44061 (local address 127.0.0.1:41690)
0926 11:29:50.337769 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:50.337782 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:50.337881 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:50.338122 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:50.338124 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:50.338541 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:29:50.338637 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:50.338642 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.339343 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.339346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:50.339955 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:50.339962 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.340362 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.340364 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:50.340421 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:50.341018 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:29:50.341030 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:29:50.342384 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:29:50.343923 (+ 1539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:50.343930 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:50.343935 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:50.344169 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:50.344414 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:50.344416 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:50.344418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:50.344518 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:50.344947 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:29:50.344952 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:29:50.345056 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 11:29:50.345145 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 11:29:50.345264 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":87,"threads_started":1}
I20260926 11:29:50.349464 1505 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41690:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 11:29:50.350210 1505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:29:50.354894 1640 tablet_service.cc:1467] Processing CreateTablet for tablet 6966e97f18124122b334632cccb5e02d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0c28d4d0c3db4dacba20446abb5536e0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 11:29:50.354969 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 6966e97f18124122b334632cccb5e02d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0c28d4d0c3db4dacba20446abb5536e0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 11:29:50.355157 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 6966e97f18124122b334632cccb5e02d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0c28d4d0c3db4dacba20446abb5536e0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 11:29:50.355321 1640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6966e97f18124122b334632cccb5e02d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:29:50.355365 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6966e97f18124122b334632cccb5e02d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:29:50.355482 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6966e97f18124122b334632cccb5e02d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:29:50.357944 2013 tablet_bootstrap.cc:492] T 6966e97f18124122b334632cccb5e02d P a9dfcdf25b434d37bea03047ca294303: Bootstrap starting.
I20260926 11:29:50.358283 2015 tablet_bootstrap.cc:492] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210: Bootstrap starting.
I20260926 11:29:50.358345 2014 tablet_bootstrap.cc:492] T 6966e97f18124122b334632cccb5e02d P e10e18126798422f819da4fc40b3a045: Bootstrap starting.
I20260926 11:29:50.358937 2013 tablet_bootstrap.cc:654] T 6966e97f18124122b334632cccb5e02d P a9dfcdf25b434d37bea03047ca294303: Neither blocks nor log segments found. Creating new log.
I20260926 11:29:50.359090 2015 tablet_bootstrap.cc:654] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210: Neither blocks nor log segments found. Creating new log.
I20260926 11:29:50.359215 2014 tablet_bootstrap.cc:654] T 6966e97f18124122b334632cccb5e02d P e10e18126798422f819da4fc40b3a045: Neither blocks nor log segments found. Creating new log.
I20260926 11:29:50.360391 2013 tablet_bootstrap.cc:492] T 6966e97f18124122b334632cccb5e02d P a9dfcdf25b434d37bea03047ca294303: No bootstrap required, opened a new log
I20260926 11:29:50.360400 2015 tablet_bootstrap.cc:492] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210: No bootstrap required, opened a new log
I20260926 11:29:50.360466 2014 tablet_bootstrap.cc:492] T 6966e97f18124122b334632cccb5e02d P e10e18126798422f819da4fc40b3a045: No bootstrap required, opened a new log
I20260926 11:29:50.360477 2015 ts_tablet_manager.cc:1397] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 11:29:50.360488 2013 ts_tablet_manager.cc:1397] T 6966e97f18124122b334632cccb5e02d P a9dfcdf25b434d37bea03047ca294303: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260926 11:29:50.360560 2014 ts_tablet_manager.cc:1397] T 6966e97f18124122b334632cccb5e02d P e10e18126798422f819da4fc40b3a045: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 11:29:50.360872 2015 raft_consensus.cc:348] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } } peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } }
I20260926 11:29:50.360872 2013 raft_consensus.cc:348] T 6966e97f18124122b334632cccb5e02d P a9dfcdf25b434d37bea03047ca294303 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } } peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } }
I20260926 11:29:50.360919 2014 raft_consensus.cc:348] T 6966e97f18124122b334632cccb5e02d P e10e18126798422f819da4fc40b3a045 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } } peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } }
I20260926 11:29:50.360997 2015 raft_consensus.cc:374] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:29:50.360997 2013 raft_consensus.cc:374] T 6966e97f18124122b334632cccb5e02d P a9dfcdf25b434d37bea03047ca294303 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:29:50.361024 2014 raft_consensus.cc:374] T 6966e97f18124122b334632cccb5e02d P e10e18126798422f819da4fc40b3a045 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:29:50.361033 2013 raft_consensus.cc:729] T 6966e97f18124122b334632cccb5e02d P a9dfcdf25b434d37bea03047ca294303 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9dfcdf25b434d37bea03047ca294303, State: Initialized, Role: FOLLOWER
I20260926 11:29:50.361033 2015 raft_consensus.cc:729] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e01082f3d2642f695a65ec8e1231210, State: Initialized, Role: FOLLOWER
I20260926 11:29:50.361063 2014 raft_consensus.cc:729] T 6966e97f18124122b334632cccb5e02d P e10e18126798422f819da4fc40b3a045 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e10e18126798422f819da4fc40b3a045, State: Initialized, Role: FOLLOWER
I20260926 11:29:50.361159 2015 consensus_queue.cc:260] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } } peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } }
I20260926 11:29:50.361159 2013 consensus_queue.cc:260] T 6966e97f18124122b334632cccb5e02d P a9dfcdf25b434d37bea03047ca294303 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } } peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } }
I20260926 11:29:50.361173 2014 consensus_queue.cc:260] T 6966e97f18124122b334632cccb5e02d P e10e18126798422f819da4fc40b3a045 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } } peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } }
I20260926 11:29:50.361404 2015 ts_tablet_manager.cc:1428] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 11:29:50.361404 2013 ts_tablet_manager.cc:1428] T 6966e97f18124122b334632cccb5e02d P a9dfcdf25b434d37bea03047ca294303: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260926 11:29:50.361466 2014 ts_tablet_manager.cc:1428] T 6966e97f18124122b334632cccb5e02d P e10e18126798422f819da4fc40b3a045: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 11:29:50.361598 2031 tablet_replica.cc:439] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } } peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } } }
I20260926 11:29:50.361722 2031 tablet_replica.cc:442] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:29:50.361711 2020 tablet_replica.cc:439] T 6966e97f18124122b334632cccb5e02d P e10e18126798422f819da4fc40b3a045: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } } peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } } }
I20260926 11:29:50.361832 2020 tablet_replica.cc:442] T 6966e97f18124122b334632cccb5e02d P e10e18126798422f819da4fc40b3a045: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:29:50.361882 2019 tablet_replica.cc:439] T 6966e97f18124122b334632cccb5e02d P a9dfcdf25b434d37bea03047ca294303: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } } peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } } }
I20260926 11:29:50.361987 2019 tablet_replica.cc:442] T 6966e97f18124122b334632cccb5e02d P a9dfcdf25b434d37bea03047ca294303: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:29:50.423285 2031 raft_consensus.cc:482] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:29:50.423431 2031 raft_consensus.cc:504] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } } peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } }
I20260926 11:29:50.423861 2031 leader_election.cc:288] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e10e18126798422f819da4fc40b3a045 (127.0.98.1:41165), a9dfcdf25b434d37bea03047ca294303 (127.0.98.2:39351)
I20260926 11:29:50.424232 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6966e97f18124122b334632cccb5e02d" candidate_uuid: "0e01082f3d2642f695a65ec8e1231210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e10e18126798422f819da4fc40b3a045" is_pre_election: true
I20260926 11:29:50.424312 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6966e97f18124122b334632cccb5e02d" candidate_uuid: "0e01082f3d2642f695a65ec8e1231210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9dfcdf25b434d37bea03047ca294303" is_pre_election: true
I20260926 11:29:50.424399 1664 raft_consensus.cc:2454] T 6966e97f18124122b334632cccb5e02d P e10e18126798422f819da4fc40b3a045 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e01082f3d2642f695a65ec8e1231210 in term 0.
I20260926 11:29:50.424437 1804 raft_consensus.cc:2454] T 6966e97f18124122b334632cccb5e02d P a9dfcdf25b434d37bea03047ca294303 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e01082f3d2642f695a65ec8e1231210 in term 0.
I20260926 11:29:50.424724 1877 leader_election.cc:302] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0e01082f3d2642f695a65ec8e1231210, a9dfcdf25b434d37bea03047ca294303, e10e18126798422f819da4fc40b3a045; no voters:
I20260926 11:29:50.424968 2031 raft_consensus.cc:2781] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:29:50.425019 2031 raft_consensus.cc:482] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:29:50.425045 2031 raft_consensus.cc:3037] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:29:50.426019 2031 raft_consensus.cc:504] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } } peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } }
I20260926 11:29:50.426374 2031 leader_election.cc:288] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210 [CANDIDATE]: Term 1 election: Requested vote from peers e10e18126798422f819da4fc40b3a045 (127.0.98.1:41165), a9dfcdf25b434d37bea03047ca294303 (127.0.98.2:39351)
I20260926 11:29:50.426709 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6966e97f18124122b334632cccb5e02d" candidate_uuid: "0e01082f3d2642f695a65ec8e1231210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e10e18126798422f819da4fc40b3a045"
I20260926 11:29:50.426741 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6966e97f18124122b334632cccb5e02d" candidate_uuid: "0e01082f3d2642f695a65ec8e1231210" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9dfcdf25b434d37bea03047ca294303"
I20260926 11:29:50.426829 1664 raft_consensus.cc:3037] T 6966e97f18124122b334632cccb5e02d P e10e18126798422f819da4fc40b3a045 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:29:50.426846 1804 raft_consensus.cc:3037] T 6966e97f18124122b334632cccb5e02d P a9dfcdf25b434d37bea03047ca294303 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:29:50.427948 1804 raft_consensus.cc:2454] T 6966e97f18124122b334632cccb5e02d P a9dfcdf25b434d37bea03047ca294303 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e01082f3d2642f695a65ec8e1231210 in term 1.
I20260926 11:29:50.427948 1664 raft_consensus.cc:2454] T 6966e97f18124122b334632cccb5e02d P e10e18126798422f819da4fc40b3a045 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e01082f3d2642f695a65ec8e1231210 in term 1.
I20260926 11:29:50.428349 1875 leader_election.cc:302] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e01082f3d2642f695a65ec8e1231210, a9dfcdf25b434d37bea03047ca294303; no voters:
I20260926 11:29:50.428574 2031 raft_consensus.cc:2781] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:29:50.428675 2031 raft_consensus.cc:686] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210 [term 1 LEADER]: Becoming Leader. State: Replica: 0e01082f3d2642f695a65ec8e1231210, State: Running, Role: LEADER
I20260926 11:29:50.428822 2031 consensus_queue.cc:237] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } } peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } }
I20260926 11:29:50.429569 2021 tablet_replica.cc:439] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210: TxnStatusTablet state changed. Reason: New leader 0e01082f3d2642f695a65ec8e1231210. Latest consensus state: current_term: 1 leader_uuid: "0e01082f3d2642f695a65ec8e1231210" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } } peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } } }
I20260926 11:29:50.429678 2021 tablet_replica.cc:442] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210: This TxnStatusTablet replica's current role is: LEADER
I20260926 11:29:50.429919 2057 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 11:29:50.430109 2057 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 11:29:50.430634 1504 catalog_manager.cc:5495] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e01082f3d2642f695a65ec8e1231210 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0e01082f3d2642f695a65ec8e1231210" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } health_report { overall_health: HEALTHY } } }
I20260926 11:29:50.444062 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:50.440660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39455 (local address 127.0.0.1:37502)
0926 11:29:50.440782 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:50.440799 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:50.440899 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:50.441176 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:50.441177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:50.441184 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 11:29:50.441266 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:50.441271 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.442251 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.442253 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:50.442836 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:50.442843 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.443381 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.443383 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:50.443445 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:50.443485 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 11:29:50.443808 (+ 323us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 11:29:50.443821 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 11:29:50.443883 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 11:29:50.443956 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 11:29:50.444090 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:50.440739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37502 (local address 127.0.98.3:39455)
0926 11:29:50.440851 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:50.440855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:50.440871 (+ 16us) server_negotiation.cc:407] Connection header received
0926 11:29:50.441019 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:50.441022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:50.441075 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:50.441156 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:29:50.441371 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.442144 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.442940 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.443282 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.443380 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:50.443601 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:29:50.443680 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:29:50.443768 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:50.443914 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 11:29:50.443966 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 11:29:50.446619 1664 raft_consensus.cc:1261] T 6966e97f18124122b334632cccb5e02d P e10e18126798422f819da4fc40b3a045 [term 1 FOLLOWER]: Refusing update from remote peer 0e01082f3d2642f695a65ec8e1231210: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:29:50.446630 1804 raft_consensus.cc:1261] T 6966e97f18124122b334632cccb5e02d P a9dfcdf25b434d37bea03047ca294303 [term 1 FOLLOWER]: Refusing update from remote peer 0e01082f3d2642f695a65ec8e1231210: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:29:50.447057 2031 consensus_queue.cc:1035] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:29:50.447233 2021 consensus_queue.cc:1035] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:29:50.448700 2020 tablet_replica.cc:439] T 6966e97f18124122b334632cccb5e02d P e10e18126798422f819da4fc40b3a045: TxnStatusTablet state changed. Reason: New leader 0e01082f3d2642f695a65ec8e1231210. Latest consensus state: current_term: 1 leader_uuid: "0e01082f3d2642f695a65ec8e1231210" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } } peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } } }
I20260926 11:29:50.448763 2019 tablet_replica.cc:439] T 6966e97f18124122b334632cccb5e02d P a9dfcdf25b434d37bea03047ca294303: TxnStatusTablet state changed. Reason: New leader 0e01082f3d2642f695a65ec8e1231210. Latest consensus state: current_term: 1 leader_uuid: "0e01082f3d2642f695a65ec8e1231210" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } } peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } } }
I20260926 11:29:50.448827 2020 tablet_replica.cc:442] T 6966e97f18124122b334632cccb5e02d P e10e18126798422f819da4fc40b3a045: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:29:50.448864 2019 tablet_replica.cc:442] T 6966e97f18124122b334632cccb5e02d P a9dfcdf25b434d37bea03047ca294303: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:29:50.450146 2021 tablet_replica.cc:439] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0e01082f3d2642f695a65ec8e1231210" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } } peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } } }
I20260926 11:29:50.450259 2021 tablet_replica.cc:442] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210: This TxnStatusTablet replica's current role is: LEADER
I20260926 11:29:50.450590 2019 tablet_replica.cc:439] T 6966e97f18124122b334632cccb5e02d P a9dfcdf25b434d37bea03047ca294303: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0e01082f3d2642f695a65ec8e1231210" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } } peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } } }
I20260926 11:29:50.450682 2019 tablet_replica.cc:442] T 6966e97f18124122b334632cccb5e02d P a9dfcdf25b434d37bea03047ca294303: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:29:50.453263 2021 tablet_replica.cc:439] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0e01082f3d2642f695a65ec8e1231210" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } } peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } } }
I20260926 11:29:50.453395 2021 tablet_replica.cc:442] T 6966e97f18124122b334632cccb5e02d P 0e01082f3d2642f695a65ec8e1231210: This TxnStatusTablet replica's current role is: LEADER
I20260926 11:29:50.454275 2020 tablet_replica.cc:439] T 6966e97f18124122b334632cccb5e02d P e10e18126798422f819da4fc40b3a045: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0e01082f3d2642f695a65ec8e1231210" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 } } peers { permanent_uuid: "a9dfcdf25b434d37bea03047ca294303" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39351 } } peers { permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 } } }
I20260926 11:29:50.454401 2020 tablet_replica.cc:442] T 6966e97f18124122b334632cccb5e02d P e10e18126798422f819da4fc40b3a045: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:29:50.460561 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:50.455876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57046 (local address 127.0.98.2:39351)
0926 11:29:50.455991 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:50.455996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:50.456171 (+ 175us) server_negotiation.cc:407] Connection header received
0926 11:29:50.456254 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:50.456257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:50.456320 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:50.456422 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:29:50.456904 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.457727 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.458850 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.459190 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.459288 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:50.459641 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:29:50.460178 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:29:50.460270 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:50.460382 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 11:29:50.460432 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 11:29:50.471138 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:50.467299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35517 (local address 127.0.98.3:39455)
0926 11:29:50.467436 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:50.467441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:50.467640 (+ 199us) server_negotiation.cc:407] Connection header received
0926 11:29:50.467721 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:50.467776 (+ 55us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:50.467880 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:50.467987 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:29:50.468272 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.468769 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.470079 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.470304 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.470419 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:50.470710 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:29:50.470774 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:29:50.470862 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:50.470944 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 11:29:50.470994 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 11:29:50.471149 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:50.467205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39455 (local address 127.0.98.2:35517)
0926 11:29:50.467528 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:50.467543 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:50.467643 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:50.467998 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:50.468002 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:50.468010 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 11:29:50.468149 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:50.468160 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.468898 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.468903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:50.469957 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:50.469966 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.470419 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.470423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:50.470527 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:50.470568 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 11:29:50.470857 (+ 289us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 11:29:50.470862 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 11:29:50.470926 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 11:29:50.471001 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":115,"threads_started":1}
I20260926 11:29:50.479099 1664 raft_consensus.cc:1261] T 8e515f325f8a466f9378deaa25f9d975 P e10e18126798422f819da4fc40b3a045 [term 1 FOLLOWER]: Refusing update from remote peer a9dfcdf25b434d37bea03047ca294303: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:29:50.479312 1944 raft_consensus.cc:1261] T 8e515f325f8a466f9378deaa25f9d975 P 0e01082f3d2642f695a65ec8e1231210 [term 1 FOLLOWER]: Refusing update from remote peer a9dfcdf25b434d37bea03047ca294303: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:29:50.479822 2044 consensus_queue.cc:1035] T 8e515f325f8a466f9378deaa25f9d975 P a9dfcdf25b434d37bea03047ca294303 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e10e18126798422f819da4fc40b3a045" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:29:50.480216 2019 consensus_queue.cc:1035] T 8e515f325f8a466f9378deaa25f9d975 P a9dfcdf25b434d37bea03047ca294303 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e01082f3d2642f695a65ec8e1231210" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:29:50.498574 2028 mvcc.cc:205] Tried to move back new op lower bound from 7333569292197289984 to 7333569291437998080. Current Snapshot: MvccSnapshot[applied={T|T < 7333569292197289984}]
I20260926 11:29:50.506950 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:50.502600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34157 (local address 127.0.98.2:39351)
0926 11:29:50.502729 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:50.502734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:50.502954 (+ 220us) server_negotiation.cc:407] Connection header received
0926 11:29:50.503237 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:50.503240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:50.503290 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:50.503364 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:29:50.503763 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.504560 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.505794 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.506000 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.506090 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:50.506470 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:29:50.506556 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:29:50.506660 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:50.506778 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 11:29:50.506825 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 11:29:50.506986 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:50.502552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39351 (local address 127.0.98.3:34157)
0926 11:29:50.502840 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:50.502854 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:50.503006 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:50.503507 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:50.503510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:50.503518 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 11:29:50.503622 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:50.503629 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.504728 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.504732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:50.505667 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:50.505676 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.506116 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.506119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:50.506205 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:50.506273 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 11:29:50.506667 (+ 394us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 11:29:50.506677 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 11:29:50.506767 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 11:29:50.506843 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":104,"threads_started":1}
I20260926 11:29:50.533550 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:29:50.533710 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:29:50.536195 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e29c70a9c58451a98b5ccf40f4b2359: Sending DeleteTablet for 3 replicas of tablet 8e515f325f8a466f9378deaa25f9d975
I20260926 11:29:50.536829 1640 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e515f325f8a466f9378deaa25f9d975 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:29:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39920
I20260926 11:29:50.537019 2038 tablet_replica.cc:330] T 8e515f325f8a466f9378deaa25f9d975 P e10e18126798422f819da4fc40b3a045: stopping tablet replica
I20260926 11:29:50.536979 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e515f325f8a466f9378deaa25f9d975 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:29:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57030
I20260926 11:29:50.537122 2038 raft_consensus.cc:2229] T 8e515f325f8a466f9378deaa25f9d975 P e10e18126798422f819da4fc40b3a045 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:29:50.537159 2036 tablet_replica.cc:330] T 8e515f325f8a466f9378deaa25f9d975 P a9dfcdf25b434d37bea03047ca294303: stopping tablet replica
I20260926 11:29:50.537241 2038 raft_consensus.cc:2258] T 8e515f325f8a466f9378deaa25f9d975 P e10e18126798422f819da4fc40b3a045 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:29:50.537288 2036 raft_consensus.cc:2229] T 8e515f325f8a466f9378deaa25f9d975 P a9dfcdf25b434d37bea03047ca294303 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:29:50.537595 2036 raft_consensus.cc:2258] T 8e515f325f8a466f9378deaa25f9d975 P a9dfcdf25b434d37bea03047ca294303 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:29:50.538331 2038 ts_tablet_manager.cc:1905] T 8e515f325f8a466f9378deaa25f9d975 P e10e18126798422f819da4fc40b3a045: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:29:50.540462 2036 ts_tablet_manager.cc:1905] T 8e515f325f8a466f9378deaa25f9d975 P a9dfcdf25b434d37bea03047ca294303: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:29:50.541213 2038 ts_tablet_manager.cc:1918] T 8e515f325f8a466f9378deaa25f9d975 P e10e18126798422f819da4fc40b3a045: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 11:29:50.541276 2038 log.cc:1196] T 8e515f325f8a466f9378deaa25f9d975 P e10e18126798422f819da4fc40b3a045: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-0/wal/wals/8e515f325f8a466f9378deaa25f9d975
I20260926 11:29:50.541531 2038 ts_tablet_manager.cc:1939] T 8e515f325f8a466f9378deaa25f9d975 P e10e18126798422f819da4fc40b3a045: Deleting consensus metadata
I20260926 11:29:50.542171 1493 catalog_manager.cc:4841] TS e10e18126798422f819da4fc40b3a045 (127.0.98.1:41165): tablet 8e515f325f8a466f9378deaa25f9d975 (table test-table [id=f321374c77494c039f58f4898ebbd72d]) successfully deleted
I20260926 11:29:50.543231 2036 ts_tablet_manager.cc:1918] T 8e515f325f8a466f9378deaa25f9d975 P a9dfcdf25b434d37bea03047ca294303: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 11:29:50.543303 2036 log.cc:1196] T 8e515f325f8a466f9378deaa25f9d975 P a9dfcdf25b434d37bea03047ca294303: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-1/wal/wals/8e515f325f8a466f9378deaa25f9d975
I20260926 11:29:50.543571 2036 ts_tablet_manager.cc:1939] T 8e515f325f8a466f9378deaa25f9d975 P a9dfcdf25b434d37bea03047ca294303: Deleting consensus metadata
I20260926 11:29:50.544231 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e515f325f8a466f9378deaa25f9d975 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:29:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37484
I20260926 11:29:50.544431 2037 tablet_replica.cc:330] T 8e515f325f8a466f9378deaa25f9d975 P 0e01082f3d2642f695a65ec8e1231210: stopping tablet replica
I20260926 11:29:50.544572 2037 raft_consensus.cc:2229] T 8e515f325f8a466f9378deaa25f9d975 P 0e01082f3d2642f695a65ec8e1231210 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:29:50.544709 2037 raft_consensus.cc:2258] T 8e515f325f8a466f9378deaa25f9d975 P 0e01082f3d2642f695a65ec8e1231210 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:29:50.545029 1491 catalog_manager.cc:4841] TS a9dfcdf25b434d37bea03047ca294303 (127.0.98.2:39351): tablet 8e515f325f8a466f9378deaa25f9d975 (table test-table [id=f321374c77494c039f58f4898ebbd72d]) successfully deleted
I20260926 11:29:50.545444 2037 ts_tablet_manager.cc:1905] T 8e515f325f8a466f9378deaa25f9d975 P 0e01082f3d2642f695a65ec8e1231210: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:29:50.547907 2037 ts_tablet_manager.cc:1918] T 8e515f325f8a466f9378deaa25f9d975 P 0e01082f3d2642f695a65ec8e1231210: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 11:29:50.547994 2037 log.cc:1196] T 8e515f325f8a466f9378deaa25f9d975 P 0e01082f3d2642f695a65ec8e1231210: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790422180210734-392-0/minicluster-data/ts-2/wal/wals/8e515f325f8a466f9378deaa25f9d975
I20260926 11:29:50.548276 2037 ts_tablet_manager.cc:1939] T 8e515f325f8a466f9378deaa25f9d975 P 0e01082f3d2642f695a65ec8e1231210: Deleting consensus metadata
I20260926 11:29:50.548869 1493 catalog_manager.cc:4841] TS 0e01082f3d2642f695a65ec8e1231210 (127.0.98.3:39455): tablet 8e515f325f8a466f9378deaa25f9d975 (table test-table [id=f321374c77494c039f58f4898ebbd72d]) successfully deleted
Sep 26 11:29:50 dist-test-slave-mrck krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422189, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 11:29:50.556653 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:50.545352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39922 (local address 127.0.98.1:41165)
0926 11:29:50.545456 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:50.545462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:50.545777 (+ 315us) server_negotiation.cc:407] Connection header received
0926 11:29:50.545996 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:50.545999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:50.546055 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:50.546141 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:50.547036 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.547822 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.549183 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.549415 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.549523 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:50.552660 (+ 3137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:50.552672 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:50.552688 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:50.552722 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:50.554739 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:50.555167 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:50.555176 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:50.555182 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:50.555280 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:50.555568 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:50.555573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:50.555575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:50.555927 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:50.556169 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:50.556333 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 11:29:50.556455 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260926 11:29:50.557103 1700 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39922
I20260926 11:29:50.566828 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:50.558828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41704 (local address 127.0.98.62:44061)
0926 11:29:50.558948 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:50.558952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:50.559438 (+ 486us) server_negotiation.cc:407] Connection header received
0926 11:29:50.559658 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:50.559661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:50.559718 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:50.559831 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:50.560522 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.561063 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.561912 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:50.562151 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:50.562258 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:50.563591 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:50.563594 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:50.563597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:50.563630 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:50.565161 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:50.565582 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:50.565587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:50.565589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:50.565673 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:50.565946 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:50.565952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:50.565953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:50.566131 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:50.566270 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:50.566513 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 11:29:50.566641 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 11:29:50.567193 1506 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41704
I20260926 11:29:50.568810 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 1579
I20260926 11:29:50.576524 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 1719
I20260926 11:29:50.583531 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 1859
I20260926 11:29:50.590488 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 1475
2026-09-26T11:29:50Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2950 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:29:50 dist-test-slave-mrck krb5kdc[2078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:29:50 dist-test-slave-mrck krb5kdc[2078](info): set up 2 sockets
Sep 26 11:29:50 dist-test-slave-mrck krb5kdc[2078](info): commencing operation
krb5kdc: starting...
W20260926 11:29:51.627590 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:29:51 dist-test-slave-mrck krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:29:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:29:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:29:51.746656 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33091
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:29:51.856460 2091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:29:51.856726 2091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:29:51.856765 2091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:29:51.860669 2091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:29:51.860724 2091 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 11:29:51.860741 2091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:29:51.860759 2091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:29:51.860774 2091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:29:51.860790 2091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:29:51.865974 2091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33091
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:29:51.866989 2091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:29:51.867978 2091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:29:51.876151 2097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:29:51.876163 2099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:29:51.876225 2091 server_base.cc:1034] running on GCE node
W20260926 11:29:51.876159 2096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:29:51.876936 2091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:29:51.878062 2091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:29:51.879243 2091 hybrid_clock.cc:648] HybridClock initialized: now 1790422191879224 us; error 30 us; skew 500 ppm
Sep 26 11:29:51 dist-test-slave-mrck krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:29:51.882346 2091 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:29:51.883186 2091 webserver.cc:460] Webserver started at http://127.0.98.62:43973/ using document root <none> and password file <none>
I20260926 11:29:51.883786 2091 fs_manager.cc:359] Metadata directory not provided
I20260926 11:29:51.883867 2091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:29:51.884043 2091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:29:51.886011 2091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "60f3b6fa425a4a4aa98939c454c03905"
format_stamp: "Formatted at 2026-09-26 11:29:51 on dist-test-slave-mrck"
server_key: "16aeb767be5e50b18554d90d0699f3ea"
server_key_iv: "b70db1c4ee219da80477736bf96b9b8b"
server_key_version: "encryptionkey@0"
I20260926 11:29:51.886562 2091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "60f3b6fa425a4a4aa98939c454c03905"
format_stamp: "Formatted at 2026-09-26 11:29:51 on dist-test-slave-mrck"
server_key: "16aeb767be5e50b18554d90d0699f3ea"
server_key_iv: "b70db1c4ee219da80477736bf96b9b8b"
server_key_version: "encryptionkey@0"
I20260926 11:29:51.890354 2091 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 11:29:51.892782 2106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:51.893982 2091 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:29:51.894129 2091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "60f3b6fa425a4a4aa98939c454c03905"
format_stamp: "Formatted at 2026-09-26 11:29:51 on dist-test-slave-mrck"
server_key: "16aeb767be5e50b18554d90d0699f3ea"
server_key_iv: "b70db1c4ee219da80477736bf96b9b8b"
server_key_version: "encryptionkey@0"
I20260926 11:29:51.894246 2091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:29:51.920605 2091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:29:51.923931 2091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:29:51.924124 2091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:29:51.931819 2091 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33091
I20260926 11:29:51.931849 2168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33091 every 8 connection(s)
I20260926 11:29:51.932739 2091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:29:51.933187 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 2091
I20260926 11:29:51.933300 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:29:51.933568 392 external_mini_cluster.cc:1428] Setting key 3c849d4d94747a9baf7ef3272cb3d9c0
I20260926 11:29:51.935844 2169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:29:51.942629 2169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60f3b6fa425a4a4aa98939c454c03905: Bootstrap starting.
Sep 26 11:29:51 dist-test-slave-mrck krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:29:51.945329 2169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60f3b6fa425a4a4aa98939c454c03905: Neither blocks nor log segments found. Creating new log.
I20260926 11:29:51.946666 2169 log.cc:824] T 00000000000000000000000000000000 P 60f3b6fa425a4a4aa98939c454c03905: Log is configured to *not* fsync() on all Append() calls
I20260926 11:29:51.948159 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:51.934933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49048 (local address 127.0.98.62:33091)
0926 11:29:51.935346 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:51.935364 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:51.935406 (+ 42us) server_negotiation.cc:407] Connection header received
0926 11:29:51.936248 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:51.936271 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:51.936679 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:51.937031 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:51.937656 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:51.939125 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:51.940046 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:51.940731 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:51.940901 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:51.943573 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:51.943578 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:51.943592 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:51.943618 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:51.945287 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:51.945814 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:51.945819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:51.945823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:51.945927 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:51.946212 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:51.946215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:51.946216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:51.946595 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:51.946730 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:51.947028 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 11:29:51.947339 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":109,"threads_started":1}
I20260926 11:29:51.948941 2169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60f3b6fa425a4a4aa98939c454c03905: No bootstrap required, opened a new log
I20260926 11:29:51.952445 2169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60f3b6fa425a4a4aa98939c454c03905 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f3b6fa425a4a4aa98939c454c03905" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33091 } }
I20260926 11:29:51.952666 2169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60f3b6fa425a4a4aa98939c454c03905 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:29:51.952750 2169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60f3b6fa425a4a4aa98939c454c03905 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60f3b6fa425a4a4aa98939c454c03905, State: Initialized, Role: FOLLOWER
I20260926 11:29:51.953147 2169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60f3b6fa425a4a4aa98939c454c03905 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f3b6fa425a4a4aa98939c454c03905" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33091 } }
I20260926 11:29:51.953287 2169 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60f3b6fa425a4a4aa98939c454c03905 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:29:51.953347 2169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60f3b6fa425a4a4aa98939c454c03905 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:29:51.953444 2169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60f3b6fa425a4a4aa98939c454c03905 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:29:51.954332 2169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60f3b6fa425a4a4aa98939c454c03905 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f3b6fa425a4a4aa98939c454c03905" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33091 } }
I20260926 11:29:51.954710 2169 leader_election.cc:302] T 00000000000000000000000000000000 P 60f3b6fa425a4a4aa98939c454c03905 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60f3b6fa425a4a4aa98939c454c03905; no voters:
I20260926 11:29:51.955034 2169 leader_election.cc:288] T 00000000000000000000000000000000 P 60f3b6fa425a4a4aa98939c454c03905 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:29:51.955145 2174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60f3b6fa425a4a4aa98939c454c03905 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:29:51.955379 2174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60f3b6fa425a4a4aa98939c454c03905 [term 1 LEADER]: Becoming Leader. State: Replica: 60f3b6fa425a4a4aa98939c454c03905, State: Running, Role: LEADER
I20260926 11:29:51.955698 2174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60f3b6fa425a4a4aa98939c454c03905 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f3b6fa425a4a4aa98939c454c03905" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33091 } }
I20260926 11:29:51.956386 2169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60f3b6fa425a4a4aa98939c454c03905 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:29:51.957521 2176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60f3b6fa425a4a4aa98939c454c03905 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60f3b6fa425a4a4aa98939c454c03905. Latest consensus state: current_term: 1 leader_uuid: "60f3b6fa425a4a4aa98939c454c03905" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f3b6fa425a4a4aa98939c454c03905" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33091 } } }
I20260926 11:29:51.957660 2176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60f3b6fa425a4a4aa98939c454c03905 [sys.catalog]: This master's current role is: LEADER
I20260926 11:29:51.958031 2183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:29:51.958006 2175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60f3b6fa425a4a4aa98939c454c03905 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60f3b6fa425a4a4aa98939c454c03905" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60f3b6fa425a4a4aa98939c454c03905" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33091 } } }
I20260926 11:29:51.958601 2175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60f3b6fa425a4a4aa98939c454c03905 [sys.catalog]: This master's current role is: LEADER
I20260926 11:29:51.961246 2183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:29:51.966677 2183 catalog_manager.cc:1347] Generated new cluster ID: 8357e67756c74ab394e785d7d925ae7a
I20260926 11:29:51.966739 2183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:29:51.977718 2183 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:29:51.978899 2183 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:29:51.984378 2183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60f3b6fa425a4a4aa98939c454c03905: Generated new TSK 0
I20260926 11:29:51.985157 2183 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:29:52.022212 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33091
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:29:52.132568 2195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:29:52.132864 2195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:29:52.132946 2195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:29:52.136945 2195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:29:52.137028 2195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:29:52.137100 2195 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 11:29:52.137146 2195 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 11:29:52.137202 2195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:29:52.142307 2195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:29:52.143354 2195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:29:52.144423 2195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:29:52.154172 2201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:29:52.154178 2200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:29:52.154322 2195 server_base.cc:1034] running on GCE node
W20260926 11:29:52.154166 2203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:29:52.154821 2195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:29:52.155441 2195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:29:52.156636 2195 hybrid_clock.cc:648] HybridClock initialized: now 1790422192156626 us; error 30 us; skew 500 ppm
Sep 26 11:29:52 dist-test-slave-mrck krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:29:52.159384 2195 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:29:52.160266 2195 webserver.cc:460] Webserver started at http://127.0.98.1:43565/ using document root <none> and password file <none>
I20260926 11:29:52.160844 2195 fs_manager.cc:359] Metadata directory not provided
I20260926 11:29:52.160909 2195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:29:52.161140 2195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:29:52.163137 2195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-0/data/instance:
uuid: "fe0b582cabc24cb39653fb7f3fbcc549"
format_stamp: "Formatted at 2026-09-26 11:29:52 on dist-test-slave-mrck"
server_key: "d37cfaa2826d2a74a9855861f4173066"
server_key_iv: "569798e5a534a54e9d9ca5bdf4093d30"
server_key_version: "encryptionkey@0"
I20260926 11:29:52.163681 2195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fe0b582cabc24cb39653fb7f3fbcc549"
format_stamp: "Formatted at 2026-09-26 11:29:52 on dist-test-slave-mrck"
server_key: "d37cfaa2826d2a74a9855861f4173066"
server_key_iv: "569798e5a534a54e9d9ca5bdf4093d30"
server_key_version: "encryptionkey@0"
I20260926 11:29:52.167217 2195 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 11:29:52.169549 2210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:52.170696 2195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:29:52.170825 2195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "fe0b582cabc24cb39653fb7f3fbcc549"
format_stamp: "Formatted at 2026-09-26 11:29:52 on dist-test-slave-mrck"
server_key: "d37cfaa2826d2a74a9855861f4173066"
server_key_iv: "569798e5a534a54e9d9ca5bdf4093d30"
server_key_version: "encryptionkey@0"
I20260926 11:29:52.170921 2195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:29:52.204973 2195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:29:52.208225 2195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:29:52.208439 2195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:29:52.209874 2195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:29:52.209921 2195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:52.209986 2195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:29:52.210019 2195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 11:29:52 dist-test-slave-mrck krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:29:52.220846 2195 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40609
I20260926 11:29:52.220942 2325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40609 every 8 connection(s)
I20260926 11:29:52.221709 2195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:29:52.224195 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:52.210816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38283 (local address 127.0.98.62:33091)
0926 11:29:52.210940 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:52.210944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:52.212081 (+ 1137us) server_negotiation.cc:407] Connection header received
0926 11:29:52.213145 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:52.213149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:52.213232 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:52.213317 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:52.215106 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.216021 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.217235 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.217483 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.217596 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:52.220755 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:52.220758 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:52.220761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:52.220789 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:52.222401 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:52.222897 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:52.222900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:52.222901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:52.223001 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:52.223248 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:52.223250 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:52.223251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:52.223406 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:52.223497 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:52.223966 (+ 469us) server_negotiation.cc:299] Negotiation successful
0926 11:29:52.224079 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 11:29:52.225036 2219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:52.211395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33091 (local address 127.0.98.1:38283)
0926 11:29:52.211936 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:52.211980 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:52.212807 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:52.213793 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:52.213801 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:52.214211 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:29:52.214933 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:52.214955 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.216137 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.216141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:52.217117 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:52.217126 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.217639 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.217643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:52.217814 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:52.218996 (+ 1182us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:29:52.219021 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:29:52.220564 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:29:52.222531 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:52.222536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:52.222551 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:52.222809 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:52.223063 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:52.223065 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:52.223066 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:52.223177 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:52.223493 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:29:52.223497 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:29:52.223705 (+ 208us) client_negotiation.cc:769] Sending connection context
0926 11:29:52.223922 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 11:29:52.224153 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":126,"threads_started":1}
I20260926 11:29:52.226285 2326 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33091
I20260926 11:29:52.226516 2326 heartbeater.cc:464] Registering TS with master...
I20260926 11:29:52.226984 2326 heartbeater.cc:511] Master 127.0.98.62:33091 requested a full tablet report, sending...
I20260926 11:29:52.228529 2123 ts_manager.cc:194] Registered new tserver with Master: fe0b582cabc24cb39653fb7f3fbcc549 (127.0.98.1:40609)
I20260926 11:29:52.229001 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 2195
I20260926 11:29:52.229113 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:29:52.229425 392 external_mini_cluster.cc:1428] Setting key f956d088a847005e83af724bde3d1a4c
I20260926 11:29:52.230124 2123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38283
I20260926 11:29:52.236164 2332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:52.228574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33091 (local address 127.0.98.1:35699)
0926 11:29:52.228902 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:52.228916 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:52.229012 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:52.229319 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:52.229323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:52.229542 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:29:52.229651 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:52.229658 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.230536 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.230539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:52.231361 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:52.231372 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.231962 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.231976 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:52.232163 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:52.232720 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:29:52.232733 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:29:52.233069 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:29:52.234653 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:52.234658 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:52.234660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:52.234851 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:52.235200 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:52.235202 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:52.235203 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:52.235274 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:52.235690 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:29:52.235692 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:29:52.235805 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 11:29:52.235895 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 11:29:52.236022 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":4,"thread_start_us":84,"threads_started":1}
I20260926 11:29:52.236359 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:52.228653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35699 (local address 127.0.98.62:33091)
0926 11:29:52.228763 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:52.228767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:52.229001 (+ 234us) server_negotiation.cc:407] Connection header received
0926 11:29:52.229160 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:52.229166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:52.229216 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:52.229302 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:52.229767 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.230413 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.231499 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.231829 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.231937 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:52.233199 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:52.233202 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:52.233205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:52.233228 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:52.234541 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:52.234988 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:52.234993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:52.234998 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:52.235098 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:52.235364 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:52.235369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:52.235371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:52.235595 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:52.235704 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:52.236005 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 11:29:52.236139 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":48}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:29:52.254387 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33091
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:29:52.364596 2335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:29:52.364848 2335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:29:52.364929 2335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:29:52.368988 2335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:29:52.369052 2335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:29:52.369093 2335 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 11:29:52.369107 2335 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 11:29:52.369212 2335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:29:52.374425 2335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:29:52.375492 2335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:29:52.376593 2335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:29:52.386025 2343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:29:52.386030 2341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:29:52.386059 2340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:29:52.386343 2335 server_base.cc:1034] running on GCE node
I20260926 11:29:52.386821 2335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:29:52.387455 2335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:29:52.388653 2335 hybrid_clock.cc:648] HybridClock initialized: now 1790422192388633 us; error 33 us; skew 500 ppm
Sep 26 11:29:52 dist-test-slave-mrck krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:29:52.391487 2335 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:29:52.392494 2335 webserver.cc:460] Webserver started at http://127.0.98.2:32833/ using document root <none> and password file <none>
I20260926 11:29:52.393060 2335 fs_manager.cc:359] Metadata directory not provided
I20260926 11:29:52.393146 2335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:29:52.393371 2335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:29:52.395378 2335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-1/data/instance:
uuid: "3af2bda673cf4d7293ad681f9d4e4432"
format_stamp: "Formatted at 2026-09-26 11:29:52 on dist-test-slave-mrck"
server_key: "b9ab2fd0663c06a857ea99d4000fa153"
server_key_iv: "036cc1ee0ba7c0871394c99a9861bc1b"
server_key_version: "encryptionkey@0"
I20260926 11:29:52.395979 2335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3af2bda673cf4d7293ad681f9d4e4432"
format_stamp: "Formatted at 2026-09-26 11:29:52 on dist-test-slave-mrck"
server_key: "b9ab2fd0663c06a857ea99d4000fa153"
server_key_iv: "036cc1ee0ba7c0871394c99a9861bc1b"
server_key_version: "encryptionkey@0"
I20260926 11:29:52.399703 2335 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 11:29:52.402123 2350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:52.403337 2335 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:29:52.403508 2335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "3af2bda673cf4d7293ad681f9d4e4432"
format_stamp: "Formatted at 2026-09-26 11:29:52 on dist-test-slave-mrck"
server_key: "b9ab2fd0663c06a857ea99d4000fa153"
server_key_iv: "036cc1ee0ba7c0871394c99a9861bc1b"
server_key_version: "encryptionkey@0"
I20260926 11:29:52.403623 2335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:29:52.426195 2335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:29:52.430032 2335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:29:52.430248 2335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:29:52.431809 2335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:29:52.431880 2335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:52.431943 2335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:29:52.431975 2335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 11:29:52 dist-test-slave-mrck krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:29:52.443113 2335 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39889
I20260926 11:29:52.443246 2465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39889 every 8 connection(s)
I20260926 11:29:52.444119 2335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:29:52.445892 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:52.432700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43093 (local address 127.0.98.62:33091)
0926 11:29:52.432867 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:52.432873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:52.433806 (+ 933us) server_negotiation.cc:407] Connection header received
0926 11:29:52.434975 (+ 1169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:52.434979 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:52.435056 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:52.435162 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:52.436563 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.437104 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.438283 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.438625 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.438758 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:52.441849 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:52.441855 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:52.441859 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:52.441892 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:52.443651 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:52.444184 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:52.444189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:52.444192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:52.444276 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:52.444710 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:52.444713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:52.444715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:52.444954 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:52.445066 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:52.445600 (+ 534us) server_negotiation.cc:299] Negotiation successful
0926 11:29:52.445725 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 11:29:52.446658 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:52.433154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33091 (local address 127.0.98.2:43093)
0926 11:29:52.433628 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:52.433681 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:52.434659 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:52.435342 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:52.435348 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:52.435702 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:29:52.436403 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:52.436421 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.437245 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.437250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:52.438142 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:52.438151 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.438803 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.438807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:52.439037 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:52.439779 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:29:52.439804 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:29:52.441663 (+ 1859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:29:52.443788 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:52.443797 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:52.443812 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:52.444078 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:52.444394 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:52.444397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:52.444399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:52.444578 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:52.445074 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:29:52.445083 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:29:52.445353 (+ 270us) client_negotiation.cc:769] Sending connection context
0926 11:29:52.445540 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 11:29:52.445858 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":141,"threads_started":1}
I20260926 11:29:52.448015 2466 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33091
I20260926 11:29:52.448186 2466 heartbeater.cc:464] Registering TS with master...
I20260926 11:29:52.448745 2466 heartbeater.cc:511] Master 127.0.98.62:33091 requested a full tablet report, sending...
I20260926 11:29:52.449780 2123 ts_manager.cc:194] Registered new tserver with Master: 3af2bda673cf4d7293ad681f9d4e4432 (127.0.98.2:39889)
I20260926 11:29:52.450577 2123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43093
I20260926 11:29:52.451061 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 2335
I20260926 11:29:52.451167 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:29:52.451447 392 external_mini_cluster.cc:1428] Setting key 938105fa4c162c827dc0b3fe2a258b79
I20260926 11:29:52.458074 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:52.450179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39865 (local address 127.0.98.62:33091)
0926 11:29:52.450297 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:52.450301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:52.450428 (+ 127us) server_negotiation.cc:407] Connection header received
0926 11:29:52.450523 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:52.450525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:52.450558 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:52.450622 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:52.451217 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.451952 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.452980 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.453342 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.453478 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:52.454751 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:52.454754 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:52.454755 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:52.454775 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:52.456163 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:52.456638 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:52.456641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:52.456642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:52.456742 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:52.457062 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:52.457065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:52.457067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:52.457423 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:52.457514 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:52.457858 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 11:29:52.457958 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 11:29:52.458576 2472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:52.450058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33091 (local address 127.0.98.2:39865)
0926 11:29:52.450338 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:52.450352 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:52.450446 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:52.450665 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:52.450669 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:52.450888 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:29:52.451035 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:52.451042 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.452058 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.452061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:52.452850 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:52.452860 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.453471 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.453473 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:52.453541 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:52.454075 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:29:52.454086 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:29:52.454618 (+ 532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:29:52.456267 (+ 1649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:52.456273 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:52.456276 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:52.456545 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:52.456840 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:52.456846 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:52.456850 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:52.456967 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:52.457528 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:29:52.457534 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:29:52.457649 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 11:29:52.457779 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 11:29:52.457968 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:29:52.478194 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33091
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:29:52.587334 2475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:29:52.587632 2475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:29:52.587713 2475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:29:52.591710 2475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:29:52.591871 2475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:29:52.591954 2475 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 11:29:52.591995 2475 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 11:29:52.592062 2475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:29:52.597486 2475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:29:52.598616 2475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:29:52.599628 2475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:29:52.609205 2481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:29:52.609205 2480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:29:52.609295 2483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:29:52.609534 2475 server_base.cc:1034] running on GCE node
I20260926 11:29:52.610016 2475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:29:52.610632 2475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:29:52.611805 2475 hybrid_clock.cc:648] HybridClock initialized: now 1790422192611767 us; error 48 us; skew 500 ppm
Sep 26 11:29:52 dist-test-slave-mrck krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:29:52.614578 2475 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:29:52.615360 2475 webserver.cc:460] Webserver started at http://127.0.98.3:38401/ using document root <none> and password file <none>
I20260926 11:29:52.615937 2475 fs_manager.cc:359] Metadata directory not provided
I20260926 11:29:52.616000 2475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:29:52.616158 2475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:29:52.618086 2475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-2/data/instance:
uuid: "e59bb6f9de19490a833d59ad8dfae247"
format_stamp: "Formatted at 2026-09-26 11:29:52 on dist-test-slave-mrck"
server_key: "c4a0eac9cc35649d64bc5f250420fd69"
server_key_iv: "c3ae5dff32e89fb2b17987a62498366d"
server_key_version: "encryptionkey@0"
I20260926 11:29:52.618597 2475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e59bb6f9de19490a833d59ad8dfae247"
format_stamp: "Formatted at 2026-09-26 11:29:52 on dist-test-slave-mrck"
server_key: "c4a0eac9cc35649d64bc5f250420fd69"
server_key_iv: "c3ae5dff32e89fb2b17987a62498366d"
server_key_version: "encryptionkey@0"
I20260926 11:29:52.622185 2475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 11:29:52.624548 2490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:52.625730 2475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:29:52.625844 2475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "e59bb6f9de19490a833d59ad8dfae247"
format_stamp: "Formatted at 2026-09-26 11:29:52 on dist-test-slave-mrck"
server_key: "c4a0eac9cc35649d64bc5f250420fd69"
server_key_iv: "c3ae5dff32e89fb2b17987a62498366d"
server_key_version: "encryptionkey@0"
I20260926 11:29:52.625965 2475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:29:52.649533 2475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:29:52.652918 2475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:29:52.653120 2475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:29:52.654562 2475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:29:52.654636 2475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:52.654711 2475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:29:52.654762 2475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 11:29:52 dist-test-slave-mrck krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422192, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:29:52.665737 2475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39985
I20260926 11:29:52.665753 2605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39985 every 8 connection(s)
I20260926 11:29:52.666630 2475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:29:52.670065 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:52.655573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35421 (local address 127.0.98.62:33091)
0926 11:29:52.655701 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:52.655705 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:52.656913 (+ 1208us) server_negotiation.cc:407] Connection header received
0926 11:29:52.658062 (+ 1149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:52.658065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:52.658125 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:52.658227 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:52.659899 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.660710 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.662570 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.662870 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.662956 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:52.666175 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:52.666178 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:52.666181 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:52.666213 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:52.668015 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:52.668545 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:52.668548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:52.668550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:52.668626 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:52.669099 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:52.669102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:52.669103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:52.669292 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:52.669373 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:52.669825 (+ 452us) server_negotiation.cc:299] Negotiation successful
0926 11:29:52.669929 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 11:29:52.670886 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:52.656168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33091 (local address 127.0.98.3:35421)
0926 11:29:52.656681 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:52.656802 (+ 121us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:52.657775 (+ 973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:52.658449 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:52.658460 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:52.658924 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:29:52.659668 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:52.659693 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.660840 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.660845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:52.662433 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:52.662444 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.663050 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.663054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:52.663269 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:52.664132 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:29:52.664160 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:29:52.666001 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:29:52.668143 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:52.668147 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:52.668158 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:52.668440 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:52.668730 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:52.668733 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:52.668734 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:52.668904 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:52.669389 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:29:52.669393 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:29:52.669616 (+ 223us) client_negotiation.cc:769] Sending connection context
0926 11:29:52.669760 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 11:29:52.670007 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":128,"threads_started":1}
I20260926 11:29:52.672194 2606 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33091
I20260926 11:29:52.672356 2606 heartbeater.cc:464] Registering TS with master...
I20260926 11:29:52.672900 2606 heartbeater.cc:511] Master 127.0.98.62:33091 requested a full tablet report, sending...
I20260926 11:29:52.673930 2122 ts_manager.cc:194] Registered new tserver with Master: e59bb6f9de19490a833d59ad8dfae247 (127.0.98.3:39985)
I20260926 11:29:52.674647 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 2475
I20260926 11:29:52.674769 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:29:52.674780 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35421
I20260926 11:29:52.675140 392 external_mini_cluster.cc:1428] Setting key ee8ac0e3e61f4eb74e96750f2e0ad743
I20260926 11:29:52.677263 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:29:52.682190 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:52.674354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57787 (local address 127.0.98.62:33091)
0926 11:29:52.674487 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:52.674491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:52.674733 (+ 242us) server_negotiation.cc:407] Connection header received
0926 11:29:52.674857 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:52.674861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:52.674900 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:52.674949 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:52.675784 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.676539 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.677543 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.677887 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.678003 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:52.679176 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:52.679179 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:52.679181 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:52.679200 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:52.680619 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:52.681039 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:52.681044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:52.681046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:52.681127 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:52.681416 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:52.681419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:52.681421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:52.681588 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:52.681690 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:52.681937 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 11:29:52.682041 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 11:29:52.682201 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:52.674265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33091 (local address 127.0.98.3:57787)
0926 11:29:52.674633 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:52.674647 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:52.674744 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:52.675119 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:52.675122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:52.675301 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:29:52.675423 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:52.675649 (+ 226us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.676808 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.676811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:52.677437 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:52.677445 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.678009 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.678012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:52.678080 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:52.678629 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:29:52.678641 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:29:52.679032 (+ 391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:29:52.680728 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:52.680733 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:52.680735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:52.680940 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:52.681222 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:52.681225 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:52.681227 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:52.681313 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:52.681678 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:29:52.681681 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:29:52.681769 (+ 88us) client_negotiation.cc:769] Sending connection context
0926 11:29:52.681885 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 11:29:52.682030 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":84,"threads_started":1}
Sep 26 11:29:52 dist-test-slave-mrck krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 11:29:52 dist-test-slave-mrck krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:29:52.703038 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:52.695041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49060 (local address 127.0.98.62:33091)
0926 11:29:52.695179 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:52.695183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:52.695194 (+ 11us) server_negotiation.cc:407] Connection header received
0926 11:29:52.695229 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:52.695231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:52.695268 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:52.695357 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:52.696108 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.696624 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.697523 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.697749 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.697856 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:52.700221 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:52.700232 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:52.700234 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:52.700255 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:52.701436 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:52.701839 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:52.701844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:52.701846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:52.701932 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:52.702201 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:52.702205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:52.702207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:52.702386 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:52.702521 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:52.702742 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 11:29:52.702863 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 11:29:52.705747 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:29:52.708892 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:29:52.724246 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:52.718574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43990 (local address 127.0.98.3:39985)
0926 11:29:52.719070 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:52.719084 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:52.719830 (+ 746us) server_negotiation.cc:407] Connection header received
0926 11:29:52.720051 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:52.720058 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:52.720245 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:52.720394 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:29:52.720593 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.721437 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.722859 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.723485 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.723609 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:52.723800 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:52.723999 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 11:29:52.724093 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":101,"threads_started":1}
I20260926 11:29:52.724234 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:52.718180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39985 (local address 127.0.0.1:43990)
0926 11:29:52.719716 (+ 1536us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:52.719755 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:52.719894 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:52.720385 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:52.720390 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:52.720411 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:29:52.720496 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:52.720501 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.721545 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.721548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:52.722719 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:52.722729 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.723607 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.723611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:52.723773 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:52.723824 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 11:29:52.723956 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 11:29:52.724087 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1463,"mutex_wait_us":29,"thread_start_us":83,"threads_started":1}
I20260926 11:29:52.724535 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:52.719043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53254 (local address 127.0.98.1:40609)
0926 11:29:52.719286 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:52.719305 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:52.719327 (+ 22us) server_negotiation.cc:407] Connection header received
0926 11:29:52.719401 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:52.719407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:52.719636 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:52.719830 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:29:52.720326 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.721183 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.722774 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.723618 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.723719 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:52.723886 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:52.724299 (+ 413us) server_negotiation.cc:299] Negotiation successful
0926 11:29:52.724393 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":89,"threads_started":1}
I20260926 11:29:52.725075 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:52.717800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40609 (local address 127.0.0.1:53254)
0926 11:29:52.718736 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:52.718788 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:52.718912 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:52.720012 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:52.720019 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:52.720042 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:29:52.720190 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:52.720198 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.721293 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.721296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:52.722654 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:52.722662 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.723952 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.723956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:52.724042 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:52.724063 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 11:29:52.724129 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 11:29:52.724196 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":758,"thread_start_us":88,"threads_started":1}
I20260926 11:29:52.725828 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:52.718506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39889 (local address 127.0.0.1:44118)
0926 11:29:52.720078 (+ 1572us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:52.720090 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:52.720175 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:52.721574 (+ 1399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:52.721577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:52.721596 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:29:52.721694 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:52.721699 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.723042 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.723045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:52.724178 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:52.724185 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.725414 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.725418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:52.725508 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:52.725529 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 11:29:52.725598 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 11:29:52.725666 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1517,"mutex_wait_us":2,"thread_start_us":83,"threads_started":1}
I20260926 11:29:52.726298 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:52.719365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44118 (local address 127.0.98.2:39889)
0926 11:29:52.720060 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:52.720071 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:52.720150 (+ 79us) server_negotiation.cc:407] Connection header received
0926 11:29:52.720288 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:52.720296 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:52.720476 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:52.720613 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:29:52.721818 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.722924 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.724301 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.725074 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.725183 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:52.725347 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:52.726065 (+ 718us) server_negotiation.cc:299] Negotiation successful
0926 11:29:52.726168 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":112,"threads_started":1}
I20260926 11:29:52.728020 2260 tablet_service.cc:1467] Processing CreateTablet for tablet aa94c72f40d54fdbb4d28d1fa782a0e3 (DEFAULT_TABLE table=test-table [id=9f6c0baf7ada448484d1e251047c434c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:29:52.728029 2540 tablet_service.cc:1467] Processing CreateTablet for tablet aa94c72f40d54fdbb4d28d1fa782a0e3 (DEFAULT_TABLE table=test-table [id=9f6c0baf7ada448484d1e251047c434c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:29:52.728579 2400 tablet_service.cc:1467] Processing CreateTablet for tablet aa94c72f40d54fdbb4d28d1fa782a0e3 (DEFAULT_TABLE table=test-table [id=9f6c0baf7ada448484d1e251047c434c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:29:52.729146 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa94c72f40d54fdbb4d28d1fa782a0e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:29:52.729142 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa94c72f40d54fdbb4d28d1fa782a0e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:29:52.729494 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa94c72f40d54fdbb4d28d1fa782a0e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:29:52.735882 2629 tablet_bootstrap.cc:492] T aa94c72f40d54fdbb4d28d1fa782a0e3 P 3af2bda673cf4d7293ad681f9d4e4432: Bootstrap starting.
I20260926 11:29:52.737008 2630 tablet_bootstrap.cc:492] T aa94c72f40d54fdbb4d28d1fa782a0e3 P fe0b582cabc24cb39653fb7f3fbcc549: Bootstrap starting.
I20260926 11:29:52.737236 2631 tablet_bootstrap.cc:492] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247: Bootstrap starting.
I20260926 11:29:52.737996 2629 tablet_bootstrap.cc:654] T aa94c72f40d54fdbb4d28d1fa782a0e3 P 3af2bda673cf4d7293ad681f9d4e4432: Neither blocks nor log segments found. Creating new log.
I20260926 11:29:52.738705 2629 log.cc:824] T aa94c72f40d54fdbb4d28d1fa782a0e3 P 3af2bda673cf4d7293ad681f9d4e4432: Log is configured to *not* fsync() on all Append() calls
I20260926 11:29:52.739121 2631 tablet_bootstrap.cc:654] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247: Neither blocks nor log segments found. Creating new log.
I20260926 11:29:52.739121 2630 tablet_bootstrap.cc:654] T aa94c72f40d54fdbb4d28d1fa782a0e3 P fe0b582cabc24cb39653fb7f3fbcc549: Neither blocks nor log segments found. Creating new log.
I20260926 11:29:52.740003 2631 log.cc:824] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247: Log is configured to *not* fsync() on all Append() calls
I20260926 11:29:52.740257 2630 log.cc:824] T aa94c72f40d54fdbb4d28d1fa782a0e3 P fe0b582cabc24cb39653fb7f3fbcc549: Log is configured to *not* fsync() on all Append() calls
I20260926 11:29:52.741016 2629 tablet_bootstrap.cc:492] T aa94c72f40d54fdbb4d28d1fa782a0e3 P 3af2bda673cf4d7293ad681f9d4e4432: No bootstrap required, opened a new log
I20260926 11:29:52.741218 2629 ts_tablet_manager.cc:1397] T aa94c72f40d54fdbb4d28d1fa782a0e3 P 3af2bda673cf4d7293ad681f9d4e4432: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260926 11:29:52.741935 2631 tablet_bootstrap.cc:492] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247: No bootstrap required, opened a new log
I20260926 11:29:52.741932 2630 tablet_bootstrap.cc:492] T aa94c72f40d54fdbb4d28d1fa782a0e3 P fe0b582cabc24cb39653fb7f3fbcc549: No bootstrap required, opened a new log
I20260926 11:29:52.742103 2630 ts_tablet_manager.cc:1397] T aa94c72f40d54fdbb4d28d1fa782a0e3 P fe0b582cabc24cb39653fb7f3fbcc549: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260926 11:29:52.742103 2631 ts_tablet_manager.cc:1397] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 11:29:52.744220 2629 raft_consensus.cc:348] T aa94c72f40d54fdbb4d28d1fa782a0e3 P 3af2bda673cf4d7293ad681f9d4e4432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:52.744464 2629 raft_consensus.cc:374] T aa94c72f40d54fdbb4d28d1fa782a0e3 P 3af2bda673cf4d7293ad681f9d4e4432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:29:52.744516 2629 raft_consensus.cc:729] T aa94c72f40d54fdbb4d28d1fa782a0e3 P 3af2bda673cf4d7293ad681f9d4e4432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3af2bda673cf4d7293ad681f9d4e4432, State: Initialized, Role: FOLLOWER
I20260926 11:29:52.744935 2629 consensus_queue.cc:260] T aa94c72f40d54fdbb4d28d1fa782a0e3 P 3af2bda673cf4d7293ad681f9d4e4432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:52.745572 2466 heartbeater.cc:503] Master 127.0.98.62:33091 was elected leader, sending a full tablet report...
I20260926 11:29:52.745935 2629 ts_tablet_manager.cc:1428] T aa94c72f40d54fdbb4d28d1fa782a0e3 P 3af2bda673cf4d7293ad681f9d4e4432: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260926 11:29:52.745925 2630 raft_consensus.cc:348] T aa94c72f40d54fdbb4d28d1fa782a0e3 P fe0b582cabc24cb39653fb7f3fbcc549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:52.745925 2631 raft_consensus.cc:348] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:52.746184 2630 raft_consensus.cc:374] T aa94c72f40d54fdbb4d28d1fa782a0e3 P fe0b582cabc24cb39653fb7f3fbcc549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:29:52.746186 2631 raft_consensus.cc:374] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:29:52.746243 2630 raft_consensus.cc:729] T aa94c72f40d54fdbb4d28d1fa782a0e3 P fe0b582cabc24cb39653fb7f3fbcc549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe0b582cabc24cb39653fb7f3fbcc549, State: Initialized, Role: FOLLOWER
I20260926 11:29:52.746249 2631 raft_consensus.cc:729] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59bb6f9de19490a833d59ad8dfae247, State: Initialized, Role: FOLLOWER
I20260926 11:29:52.746701 2631 consensus_queue.cc:260] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:52.746701 2630 consensus_queue.cc:260] T aa94c72f40d54fdbb4d28d1fa782a0e3 P fe0b582cabc24cb39653fb7f3fbcc549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:52.747601 2630 ts_tablet_manager.cc:1428] T aa94c72f40d54fdbb4d28d1fa782a0e3 P fe0b582cabc24cb39653fb7f3fbcc549: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260926 11:29:52.747601 2631 ts_tablet_manager.cc:1428] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 11:29:52.747871 2606 heartbeater.cc:503] Master 127.0.98.62:33091 was elected leader, sending a full tablet report...
I20260926 11:29:52.748039 2326 heartbeater.cc:503] Master 127.0.98.62:33091 was elected leader, sending a full tablet report...
I20260926 11:29:52.847167 2637 raft_consensus.cc:482] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:29:52.847383 2637 raft_consensus.cc:504] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:52.848742 2637 leader_election.cc:288] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe0b582cabc24cb39653fb7f3fbcc549 (127.0.98.1:40609), 3af2bda673cf4d7293ad681f9d4e4432 (127.0.98.2:39889)
I20260926 11:29:52.852972 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:52.849119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43195 (local address 127.0.98.1:40609)
0926 11:29:52.849279 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:52.849286 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:52.849349 (+ 63us) server_negotiation.cc:407] Connection header received
0926 11:29:52.849552 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:52.849555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:52.849618 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:52.849689 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:29:52.850151 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.850769 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.851916 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.852494 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.852580 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:52.852628 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:52.852823 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 11:29:52.852873 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 11:29:52.853019 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:52.849037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40609 (local address 127.0.98.3:43195)
0926 11:29:52.849265 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:52.849279 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:52.849406 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:52.849811 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:52.849813 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:52.849828 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:29:52.850013 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:52.850020 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.850878 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.850882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:52.851784 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:52.851794 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.852599 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.852602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:52.852704 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:52.852729 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 11:29:52.852803 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 11:29:52.852870 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":6,"spinlock_wait_cycles":23808}
I20260926 11:29:52.853683 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:52.849566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58809 (local address 127.0.98.2:39889)
0926 11:29:52.849712 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:52.849717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:52.849871 (+ 154us) server_negotiation.cc:407] Connection header received
0926 11:29:52.850002 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:52.850005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:52.850061 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:52.850139 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:29:52.850389 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.850914 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.852488 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.853064 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.853169 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:52.853234 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:52.853520 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 11:29:52.853581 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 11:29:52.853704 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:52.849380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39889 (local address 127.0.98.3:58809)
0926 11:29:52.849779 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:52.849789 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:52.849890 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:52.850159 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:52.850162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:52.850179 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:29:52.850287 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:52.850293 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.851018 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.851021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:52.852363 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:52.852372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.853293 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.853299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:52.853391 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:52.853408 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 11:29:52.853506 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 11:29:52.853566 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":24,"thread_start_us":153,"threads_started":1}
I20260926 11:29:52.853726 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa94c72f40d54fdbb4d28d1fa782a0e3" candidate_uuid: "e59bb6f9de19490a833d59ad8dfae247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" is_pre_election: true
I20260926 11:29:52.854106 2280 raft_consensus.cc:2454] T aa94c72f40d54fdbb4d28d1fa782a0e3 P fe0b582cabc24cb39653fb7f3fbcc549 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59bb6f9de19490a833d59ad8dfae247 in term 0.
I20260926 11:29:52.854341 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa94c72f40d54fdbb4d28d1fa782a0e3" candidate_uuid: "e59bb6f9de19490a833d59ad8dfae247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3af2bda673cf4d7293ad681f9d4e4432" is_pre_election: true
I20260926 11:29:52.854568 2491 leader_election.cc:302] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e59bb6f9de19490a833d59ad8dfae247, fe0b582cabc24cb39653fb7f3fbcc549; no voters:
I20260926 11:29:52.854622 2420 raft_consensus.cc:2454] T aa94c72f40d54fdbb4d28d1fa782a0e3 P 3af2bda673cf4d7293ad681f9d4e4432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59bb6f9de19490a833d59ad8dfae247 in term 0.
I20260926 11:29:52.854851 2637 raft_consensus.cc:2781] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:29:52.854971 2637 raft_consensus.cc:482] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:29:52.855037 2637 raft_consensus.cc:3037] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:29:52.856180 2637 raft_consensus.cc:504] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:52.856506 2637 leader_election.cc:288] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247 [CANDIDATE]: Term 1 election: Requested vote from peers fe0b582cabc24cb39653fb7f3fbcc549 (127.0.98.1:40609), 3af2bda673cf4d7293ad681f9d4e4432 (127.0.98.2:39889)
I20260926 11:29:52.856844 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa94c72f40d54fdbb4d28d1fa782a0e3" candidate_uuid: "e59bb6f9de19490a833d59ad8dfae247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe0b582cabc24cb39653fb7f3fbcc549"
I20260926 11:29:52.856873 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa94c72f40d54fdbb4d28d1fa782a0e3" candidate_uuid: "e59bb6f9de19490a833d59ad8dfae247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3af2bda673cf4d7293ad681f9d4e4432"
I20260926 11:29:52.856972 2280 raft_consensus.cc:3037] T aa94c72f40d54fdbb4d28d1fa782a0e3 P fe0b582cabc24cb39653fb7f3fbcc549 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:29:52.856987 2420 raft_consensus.cc:3037] T aa94c72f40d54fdbb4d28d1fa782a0e3 P 3af2bda673cf4d7293ad681f9d4e4432 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:29:52.858289 2280 raft_consensus.cc:2454] T aa94c72f40d54fdbb4d28d1fa782a0e3 P fe0b582cabc24cb39653fb7f3fbcc549 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59bb6f9de19490a833d59ad8dfae247 in term 1.
I20260926 11:29:52.858289 2420 raft_consensus.cc:2454] T aa94c72f40d54fdbb4d28d1fa782a0e3 P 3af2bda673cf4d7293ad681f9d4e4432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59bb6f9de19490a833d59ad8dfae247 in term 1.
I20260926 11:29:52.858608 2491 leader_election.cc:302] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3af2bda673cf4d7293ad681f9d4e4432, e59bb6f9de19490a833d59ad8dfae247; no voters:
I20260926 11:29:52.858757 2637 raft_consensus.cc:2781] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:29:52.858956 2637 raft_consensus.cc:686] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247 [term 1 LEADER]: Becoming Leader. State: Replica: e59bb6f9de19490a833d59ad8dfae247, State: Running, Role: LEADER
I20260926 11:29:52.859308 2637 consensus_queue.cc:237] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:52.862246 2121 catalog_manager.cc:5495] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247 reported cstate change: term changed from 0 to 1, leader changed from <none> to e59bb6f9de19490a833d59ad8dfae247 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e59bb6f9de19490a833d59ad8dfae247" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } health_report { overall_health: HEALTHY } } }
Sep 26 11:29:52 dist-test-slave-mrck krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:29:52.902146 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:52.893525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33091 (local address 127.0.0.1:49068)
0926 11:29:52.893829 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:52.893843 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:52.893950 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:52.894197 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:52.894199 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:52.894637 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:29:52.894738 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:52.894743 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.895465 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.895468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:52.896286 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:52.896295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.896696 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.896699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:52.896758 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:52.897373 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:29:52.897385 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:29:52.898872 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:29:52.900549 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:52.900557 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:52.900562 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:52.900785 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:52.901110 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:52.901113 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:52.901115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:52.901191 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:52.901646 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:29:52.901654 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:29:52.901788 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 11:29:52.901902 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 11:29:52.902015 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":124,"threads_started":1}
I20260926 11:29:52.902189 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:52.893600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49068 (local address 127.0.98.62:33091)
0926 11:29:52.893741 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:52.893746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:52.893925 (+ 179us) server_negotiation.cc:407] Connection header received
0926 11:29:52.894031 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:52.894033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:52.894080 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:52.894151 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:52.894835 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.895346 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.896394 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:52.896598 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:52.896679 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:52.899016 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:52.899019 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:52.899021 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:52.899041 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:52.900436 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:52.900898 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:52.900901 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:52.900903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:52.901001 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:52.901273 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:52.901275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:52.901276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:52.901468 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:52.901554 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:52.901989 (+ 435us) server_negotiation.cc:299] Negotiation successful
0926 11:29:52.902082 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 11:29:52.906049 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49068:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 11:29:52.906842 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:29:52.911580 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 50d8cbf2313849d5988f4306d23a1fad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4746438318974d2b81a86c2428ff2793]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 11:29:52.911645 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 50d8cbf2313849d5988f4306d23a1fad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4746438318974d2b81a86c2428ff2793]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 11:29:52.911567 2400 tablet_service.cc:1467] Processing CreateTablet for tablet 50d8cbf2313849d5988f4306d23a1fad (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4746438318974d2b81a86c2428ff2793]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 11:29:52.912323 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50d8cbf2313849d5988f4306d23a1fad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:29:52.912323 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50d8cbf2313849d5988f4306d23a1fad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:29:52.912328 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50d8cbf2313849d5988f4306d23a1fad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:29:52.914717 2630 tablet_bootstrap.cc:492] T 50d8cbf2313849d5988f4306d23a1fad P fe0b582cabc24cb39653fb7f3fbcc549: Bootstrap starting.
I20260926 11:29:52.915258 2631 tablet_bootstrap.cc:492] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247: Bootstrap starting.
I20260926 11:29:52.915428 2629 tablet_bootstrap.cc:492] T 50d8cbf2313849d5988f4306d23a1fad P 3af2bda673cf4d7293ad681f9d4e4432: Bootstrap starting.
I20260926 11:29:52.915848 2630 tablet_bootstrap.cc:654] T 50d8cbf2313849d5988f4306d23a1fad P fe0b582cabc24cb39653fb7f3fbcc549: Neither blocks nor log segments found. Creating new log.
I20260926 11:29:52.916294 2631 tablet_bootstrap.cc:654] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247: Neither blocks nor log segments found. Creating new log.
I20260926 11:29:52.916416 2629 tablet_bootstrap.cc:654] T 50d8cbf2313849d5988f4306d23a1fad P 3af2bda673cf4d7293ad681f9d4e4432: Neither blocks nor log segments found. Creating new log.
I20260926 11:29:52.917400 2630 tablet_bootstrap.cc:492] T 50d8cbf2313849d5988f4306d23a1fad P fe0b582cabc24cb39653fb7f3fbcc549: No bootstrap required, opened a new log
I20260926 11:29:52.917490 2630 ts_tablet_manager.cc:1397] T 50d8cbf2313849d5988f4306d23a1fad P fe0b582cabc24cb39653fb7f3fbcc549: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260926 11:29:52.917598 2631 tablet_bootstrap.cc:492] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247: No bootstrap required, opened a new log
I20260926 11:29:52.917693 2631 ts_tablet_manager.cc:1397] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
W20260926 11:29:52.917810 2607 tablet.cc:2369] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:29:52.917984 2630 raft_consensus.cc:348] T 50d8cbf2313849d5988f4306d23a1fad P fe0b582cabc24cb39653fb7f3fbcc549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:52.918133 2630 raft_consensus.cc:374] T 50d8cbf2313849d5988f4306d23a1fad P fe0b582cabc24cb39653fb7f3fbcc549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:29:52.918111 2631 raft_consensus.cc:348] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:52.918171 2630 raft_consensus.cc:729] T 50d8cbf2313849d5988f4306d23a1fad P fe0b582cabc24cb39653fb7f3fbcc549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe0b582cabc24cb39653fb7f3fbcc549, State: Initialized, Role: FOLLOWER
I20260926 11:29:52.918205 2631 raft_consensus.cc:374] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:29:52.918251 2631 raft_consensus.cc:729] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59bb6f9de19490a833d59ad8dfae247, State: Initialized, Role: FOLLOWER
I20260926 11:29:52.918295 2630 consensus_queue.cc:260] T 50d8cbf2313849d5988f4306d23a1fad P fe0b582cabc24cb39653fb7f3fbcc549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:52.918382 2631 consensus_queue.cc:260] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:52.918565 2630 ts_tablet_manager.cc:1428] T 50d8cbf2313849d5988f4306d23a1fad P fe0b582cabc24cb39653fb7f3fbcc549: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 11:29:52.918668 2631 ts_tablet_manager.cc:1428] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 11:29:52.918646 2637 tablet_replica.cc:439] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } } }
I20260926 11:29:52.918762 2637 tablet_replica.cc:442] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:29:52.918761 2636 tablet_replica.cc:439] T 50d8cbf2313849d5988f4306d23a1fad P fe0b582cabc24cb39653fb7f3fbcc549: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } } }
I20260926 11:29:52.918884 2636 tablet_replica.cc:442] T 50d8cbf2313849d5988f4306d23a1fad P fe0b582cabc24cb39653fb7f3fbcc549: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:29:52.919180 2629 tablet_bootstrap.cc:492] T 50d8cbf2313849d5988f4306d23a1fad P 3af2bda673cf4d7293ad681f9d4e4432: No bootstrap required, opened a new log
I20260926 11:29:52.919256 2629 ts_tablet_manager.cc:1397] T 50d8cbf2313849d5988f4306d23a1fad P 3af2bda673cf4d7293ad681f9d4e4432: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 11:29:52.919687 2629 raft_consensus.cc:348] T 50d8cbf2313849d5988f4306d23a1fad P 3af2bda673cf4d7293ad681f9d4e4432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:52.919839 2629 raft_consensus.cc:374] T 50d8cbf2313849d5988f4306d23a1fad P 3af2bda673cf4d7293ad681f9d4e4432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:29:52.919891 2629 raft_consensus.cc:729] T 50d8cbf2313849d5988f4306d23a1fad P 3af2bda673cf4d7293ad681f9d4e4432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3af2bda673cf4d7293ad681f9d4e4432, State: Initialized, Role: FOLLOWER
I20260926 11:29:52.920015 2629 consensus_queue.cc:260] T 50d8cbf2313849d5988f4306d23a1fad P 3af2bda673cf4d7293ad681f9d4e4432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:52.920313 2635 tablet_replica.cc:439] T 50d8cbf2313849d5988f4306d23a1fad P 3af2bda673cf4d7293ad681f9d4e4432: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } } }
I20260926 11:29:52.920464 2635 tablet_replica.cc:442] T 50d8cbf2313849d5988f4306d23a1fad P 3af2bda673cf4d7293ad681f9d4e4432: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:29:52.920331 2629 ts_tablet_manager.cc:1428] T 50d8cbf2313849d5988f4306d23a1fad P 3af2bda673cf4d7293ad681f9d4e4432: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260926 11:29:52.945485 2467 tablet.cc:2369] T 50d8cbf2313849d5988f4306d23a1fad P 3af2bda673cf4d7293ad681f9d4e4432: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 11:29:52.973158 2327 tablet.cc:2369] T 50d8cbf2313849d5988f4306d23a1fad P fe0b582cabc24cb39653fb7f3fbcc549: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:29:52.999953 2637 raft_consensus.cc:482] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:29:53.000093 2637 raft_consensus.cc:504] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:53.000527 2637 leader_election.cc:288] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe0b582cabc24cb39653fb7f3fbcc549 (127.0.98.1:40609), 3af2bda673cf4d7293ad681f9d4e4432 (127.0.98.2:39889)
I20260926 11:29:53.001063 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d8cbf2313849d5988f4306d23a1fad" candidate_uuid: "e59bb6f9de19490a833d59ad8dfae247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" is_pre_election: true
I20260926 11:29:53.001096 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d8cbf2313849d5988f4306d23a1fad" candidate_uuid: "e59bb6f9de19490a833d59ad8dfae247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3af2bda673cf4d7293ad681f9d4e4432" is_pre_election: true
I20260926 11:29:53.001230 2280 raft_consensus.cc:2454] T 50d8cbf2313849d5988f4306d23a1fad P fe0b582cabc24cb39653fb7f3fbcc549 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59bb6f9de19490a833d59ad8dfae247 in term 0.
I20260926 11:29:53.001238 2420 raft_consensus.cc:2454] T 50d8cbf2313849d5988f4306d23a1fad P 3af2bda673cf4d7293ad681f9d4e4432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59bb6f9de19490a833d59ad8dfae247 in term 0.
I20260926 11:29:53.001551 2491 leader_election.cc:302] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3af2bda673cf4d7293ad681f9d4e4432, e59bb6f9de19490a833d59ad8dfae247; no voters:
I20260926 11:29:53.001755 2637 raft_consensus.cc:2781] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:29:53.001821 2637 raft_consensus.cc:482] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:29:53.001848 2637 raft_consensus.cc:3037] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:29:53.002849 2637 raft_consensus.cc:504] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:53.003170 2637 leader_election.cc:288] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247 [CANDIDATE]: Term 1 election: Requested vote from peers fe0b582cabc24cb39653fb7f3fbcc549 (127.0.98.1:40609), 3af2bda673cf4d7293ad681f9d4e4432 (127.0.98.2:39889)
I20260926 11:29:53.003605 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d8cbf2313849d5988f4306d23a1fad" candidate_uuid: "e59bb6f9de19490a833d59ad8dfae247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe0b582cabc24cb39653fb7f3fbcc549"
I20260926 11:29:53.003621 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50d8cbf2313849d5988f4306d23a1fad" candidate_uuid: "e59bb6f9de19490a833d59ad8dfae247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3af2bda673cf4d7293ad681f9d4e4432"
I20260926 11:29:53.003762 2280 raft_consensus.cc:3037] T 50d8cbf2313849d5988f4306d23a1fad P fe0b582cabc24cb39653fb7f3fbcc549 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:29:53.003780 2420 raft_consensus.cc:3037] T 50d8cbf2313849d5988f4306d23a1fad P 3af2bda673cf4d7293ad681f9d4e4432 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:29:53.004629 2420 raft_consensus.cc:2454] T 50d8cbf2313849d5988f4306d23a1fad P 3af2bda673cf4d7293ad681f9d4e4432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59bb6f9de19490a833d59ad8dfae247 in term 1.
I20260926 11:29:53.004706 2280 raft_consensus.cc:2454] T 50d8cbf2313849d5988f4306d23a1fad P fe0b582cabc24cb39653fb7f3fbcc549 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59bb6f9de19490a833d59ad8dfae247 in term 1.
I20260926 11:29:53.004904 2491 leader_election.cc:302] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3af2bda673cf4d7293ad681f9d4e4432, e59bb6f9de19490a833d59ad8dfae247; no voters:
I20260926 11:29:53.005052 2637 raft_consensus.cc:2781] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:29:53.005157 2637 raft_consensus.cc:686] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247 [term 1 LEADER]: Becoming Leader. State: Replica: e59bb6f9de19490a833d59ad8dfae247, State: Running, Role: LEADER
I20260926 11:29:53.005290 2637 consensus_queue.cc:237] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:53.006043 2639 tablet_replica.cc:439] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247: TxnStatusTablet state changed. Reason: New leader e59bb6f9de19490a833d59ad8dfae247. Latest consensus state: current_term: 1 leader_uuid: "e59bb6f9de19490a833d59ad8dfae247" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } } }
I20260926 11:29:53.006142 2639 tablet_replica.cc:442] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247: This TxnStatusTablet replica's current role is: LEADER
I20260926 11:29:53.006536 2651 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 11:29:53.006764 2651 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 11:29:53.007190 2121 catalog_manager.cc:5495] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247 reported cstate change: term changed from 0 to 1, leader changed from <none> to e59bb6f9de19490a833d59ad8dfae247 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e59bb6f9de19490a833d59ad8dfae247" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } health_report { overall_health: HEALTHY } } }
I20260926 11:29:53.022495 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:53.018390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43998 (local address 127.0.98.3:39985)
0926 11:29:53.018517 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:53.018520 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:53.018578 (+ 58us) server_negotiation.cc:407] Connection header received
0926 11:29:53.018804 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:53.018806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:53.018871 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:53.018956 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:29:53.019252 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:53.019821 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:53.021011 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:53.021216 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:53.021316 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:53.021683 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:29:53.022097 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:29:53.022194 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:53.022327 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 11:29:53.022374 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 11:29:53.022562 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:53.018250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39985 (local address 127.0.0.1:43998)
0926 11:29:53.018509 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:53.018526 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:53.018648 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:53.019001 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:53.019005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:53.019014 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 11:29:53.019134 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:53.019141 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:53.019937 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:53.019941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:53.020872 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:53.020883 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:53.021340 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:53.021345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:53.021447 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:53.021504 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 11:29:53.022204 (+ 700us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 11:29:53.022220 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 11:29:53.022309 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 11:29:53.022408 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152}
I20260926 11:29:53.028115 2280 raft_consensus.cc:1261] T 50d8cbf2313849d5988f4306d23a1fad P fe0b582cabc24cb39653fb7f3fbcc549 [term 1 FOLLOWER]: Refusing update from remote peer e59bb6f9de19490a833d59ad8dfae247: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:29:53.028115 2420 raft_consensus.cc:1261] T 50d8cbf2313849d5988f4306d23a1fad P 3af2bda673cf4d7293ad681f9d4e4432 [term 1 FOLLOWER]: Refusing update from remote peer e59bb6f9de19490a833d59ad8dfae247: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:29:53.028786 2639 consensus_queue.cc:1035] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:29:53.029237 2639 consensus_queue.cc:1035] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:29:53.032532 2635 tablet_replica.cc:439] T 50d8cbf2313849d5988f4306d23a1fad P 3af2bda673cf4d7293ad681f9d4e4432: TxnStatusTablet state changed. Reason: New leader e59bb6f9de19490a833d59ad8dfae247. Latest consensus state: current_term: 1 leader_uuid: "e59bb6f9de19490a833d59ad8dfae247" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } } }
I20260926 11:29:53.032649 2635 tablet_replica.cc:442] T 50d8cbf2313849d5988f4306d23a1fad P 3af2bda673cf4d7293ad681f9d4e4432: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:29:53.033388 2636 tablet_replica.cc:439] T 50d8cbf2313849d5988f4306d23a1fad P fe0b582cabc24cb39653fb7f3fbcc549: TxnStatusTablet state changed. Reason: New leader e59bb6f9de19490a833d59ad8dfae247. Latest consensus state: current_term: 1 leader_uuid: "e59bb6f9de19490a833d59ad8dfae247" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } } }
I20260926 11:29:53.033864 2636 tablet_replica.cc:442] T 50d8cbf2313849d5988f4306d23a1fad P fe0b582cabc24cb39653fb7f3fbcc549: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:29:53.037854 2635 tablet_replica.cc:439] T 50d8cbf2313849d5988f4306d23a1fad P 3af2bda673cf4d7293ad681f9d4e4432: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e59bb6f9de19490a833d59ad8dfae247" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } } }
I20260926 11:29:53.037978 2635 tablet_replica.cc:442] T 50d8cbf2313849d5988f4306d23a1fad P 3af2bda673cf4d7293ad681f9d4e4432: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:29:53.038019 2636 tablet_replica.cc:439] T 50d8cbf2313849d5988f4306d23a1fad P fe0b582cabc24cb39653fb7f3fbcc549: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e59bb6f9de19490a833d59ad8dfae247" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } } }
I20260926 11:29:53.038129 2636 tablet_replica.cc:442] T 50d8cbf2313849d5988f4306d23a1fad P fe0b582cabc24cb39653fb7f3fbcc549: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:29:53.039029 2639 tablet_replica.cc:439] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e59bb6f9de19490a833d59ad8dfae247" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } } }
I20260926 11:29:53.039141 2639 tablet_replica.cc:442] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247: This TxnStatusTablet replica's current role is: LEADER
I20260926 11:29:53.039364 2654 mvcc.cc:205] Tried to move back new op lower bound from 7333569302635925504 to 7333569302551007232. Current Snapshot: MvccSnapshot[applied={T|T < 7333569302635925504}]
I20260926 11:29:53.038774 2659 tablet_replica.cc:439] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e59bb6f9de19490a833d59ad8dfae247" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } } }
I20260926 11:29:53.039552 2659 tablet_replica.cc:442] T 50d8cbf2313849d5988f4306d23a1fad P e59bb6f9de19490a833d59ad8dfae247: This TxnStatusTablet replica's current role is: LEADER
I20260926 11:29:53.039891 2655 mvcc.cc:205] Tried to move back new op lower bound from 7333569302635925504 to 7333569302551007232. Current Snapshot: MvccSnapshot[applied={T|T < 7333569302635925504}]
I20260926 11:29:53.049352 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:53.046178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44006 (local address 127.0.98.3:39985)
0926 11:29:53.046307 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:53.046312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:53.046331 (+ 19us) server_negotiation.cc:407] Connection header received
0926 11:29:53.046464 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:53.046467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:53.046517 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:53.046598 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:29:53.046856 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:53.047354 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:53.048263 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:53.048526 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:53.048636 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:53.048899 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:29:53.048986 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:29:53.049066 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:53.049166 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 11:29:53.049216 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 11:29:53.060286 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:53.057043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39985 (local address 127.0.98.3:45409)
0926 11:29:53.057196 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:53.057212 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:53.057305 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:53.057558 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:53.057561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:53.057567 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 11:29:53.057705 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:53.057711 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:53.058429 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:53.058434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:53.059062 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:53.059069 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:53.059487 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:53.059491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:53.059595 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:53.059642 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 11:29:53.059976 (+ 334us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 11:29:53.059982 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 11:29:53.060053 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 11:29:53.060149 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":25}
I20260926 11:29:53.060286 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:53.057105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45409 (local address 127.0.98.3:39985)
0926 11:29:53.057193 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:53.057197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:53.057256 (+ 59us) server_negotiation.cc:407] Connection header received
0926 11:29:53.057409 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:53.057433 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:53.057476 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:53.057553 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:29:53.057828 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:53.058321 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:53.059171 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:53.059378 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:53.059481 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:53.059827 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:29:53.059895 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:29:53.059975 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:53.060062 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 11:29:53.060124 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":41}
I20260926 11:29:53.067476 2420 raft_consensus.cc:1261] T aa94c72f40d54fdbb4d28d1fa782a0e3 P 3af2bda673cf4d7293ad681f9d4e4432 [term 1 FOLLOWER]: Refusing update from remote peer e59bb6f9de19490a833d59ad8dfae247: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:29:53.067500 2280 raft_consensus.cc:1261] T aa94c72f40d54fdbb4d28d1fa782a0e3 P fe0b582cabc24cb39653fb7f3fbcc549 [term 1 FOLLOWER]: Refusing update from remote peer e59bb6f9de19490a833d59ad8dfae247: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:29:53.067895 2659 consensus_queue.cc:1035] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:29:53.068115 2637 consensus_queue.cc:1035] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:29:53.073165 2656 mvcc.cc:205] Tried to move back new op lower bound from 7333569302800994304 to 7333569301954457600. Current Snapshot: MvccSnapshot[applied={T|T < 7333569302800994304}]
I20260926 11:29:53.113505 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:29:53.113699 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:29:53.116214 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 60f3b6fa425a4a4aa98939c454c03905: Sending DeleteTablet for 3 replicas of tablet aa94c72f40d54fdbb4d28d1fa782a0e3
I20260926 11:29:53.117009 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet aa94c72f40d54fdbb4d28d1fa782a0e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:29:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43990
I20260926 11:29:53.116999 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet aa94c72f40d54fdbb4d28d1fa782a0e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:29:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53254
I20260926 11:29:53.117149 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet aa94c72f40d54fdbb4d28d1fa782a0e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:29:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44118
I20260926 11:29:53.117664 2671 tablet_replica.cc:330] T aa94c72f40d54fdbb4d28d1fa782a0e3 P fe0b582cabc24cb39653fb7f3fbcc549: stopping tablet replica
I20260926 11:29:53.118000 2670 tablet_replica.cc:330] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247: stopping tablet replica
I20260926 11:29:53.118039 2671 raft_consensus.cc:2229] T aa94c72f40d54fdbb4d28d1fa782a0e3 P fe0b582cabc24cb39653fb7f3fbcc549 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:29:53.118121 2121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49060:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 11:29:53.118238 2671 raft_consensus.cc:2258] T aa94c72f40d54fdbb4d28d1fa782a0e3 P fe0b582cabc24cb39653fb7f3fbcc549 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:29:53.118256 2670 raft_consensus.cc:2229] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247 [term 1 LEADER]: Raft consensus shutting down.
W20260926 11:29:53.118551 2121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:29:53.118619 2672 tablet_replica.cc:330] T aa94c72f40d54fdbb4d28d1fa782a0e3 P 3af2bda673cf4d7293ad681f9d4e4432: stopping tablet replica
I20260926 11:29:53.118623 2670 raft_consensus.cc:2258] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:29:53.119066 2672 raft_consensus.cc:2229] T aa94c72f40d54fdbb4d28d1fa782a0e3 P 3af2bda673cf4d7293ad681f9d4e4432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:29:53.119253 2672 raft_consensus.cc:2258] T aa94c72f40d54fdbb4d28d1fa782a0e3 P 3af2bda673cf4d7293ad681f9d4e4432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:29:53.120523 2671 ts_tablet_manager.cc:1905] T aa94c72f40d54fdbb4d28d1fa782a0e3 P fe0b582cabc24cb39653fb7f3fbcc549: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:29:53.120523 2672 ts_tablet_manager.cc:1905] T aa94c72f40d54fdbb4d28d1fa782a0e3 P 3af2bda673cf4d7293ad681f9d4e4432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:29:53.123535 2672 ts_tablet_manager.cc:1918] T aa94c72f40d54fdbb4d28d1fa782a0e3 P 3af2bda673cf4d7293ad681f9d4e4432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 11:29:53.123534 2671 ts_tablet_manager.cc:1918] T aa94c72f40d54fdbb4d28d1fa782a0e3 P fe0b582cabc24cb39653fb7f3fbcc549: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 11:29:53.123637 2672 log.cc:1196] T aa94c72f40d54fdbb4d28d1fa782a0e3 P 3af2bda673cf4d7293ad681f9d4e4432: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-1/wal/wals/aa94c72f40d54fdbb4d28d1fa782a0e3
I20260926 11:29:53.123718 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 1cdce9d5c7254c08921f71f7cbd1fc7b (DEFAULT_TABLE table=test-table [id=64dc9f7b32b6418392f5680ef802c2c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:29:53.124027 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cdce9d5c7254c08921f71f7cbd1fc7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:29:53.124116 2672 ts_tablet_manager.cc:1939] T aa94c72f40d54fdbb4d28d1fa782a0e3 P 3af2bda673cf4d7293ad681f9d4e4432: Deleting consensus metadata
I20260926 11:29:53.124244 2671 log.cc:1196] T aa94c72f40d54fdbb4d28d1fa782a0e3 P fe0b582cabc24cb39653fb7f3fbcc549: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-0/wal/wals/aa94c72f40d54fdbb4d28d1fa782a0e3
I20260926 11:29:53.124612 2671 ts_tablet_manager.cc:1939] T aa94c72f40d54fdbb4d28d1fa782a0e3 P fe0b582cabc24cb39653fb7f3fbcc549: Deleting consensus metadata
I20260926 11:29:53.125918 2107 catalog_manager.cc:4841] TS fe0b582cabc24cb39653fb7f3fbcc549 (127.0.98.1:40609): tablet aa94c72f40d54fdbb4d28d1fa782a0e3 (table test-table [id=9f6c0baf7ada448484d1e251047c434c]) successfully deleted
I20260926 11:29:53.126101 2107 catalog_manager.cc:4841] TS 3af2bda673cf4d7293ad681f9d4e4432 (127.0.98.2:39889): tablet aa94c72f40d54fdbb4d28d1fa782a0e3 (table test-table [id=9f6c0baf7ada448484d1e251047c434c]) successfully deleted
I20260926 11:29:53.126142 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 1cdce9d5c7254c08921f71f7cbd1fc7b (DEFAULT_TABLE table=test-table [id=64dc9f7b32b6418392f5680ef802c2c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:29:53.126386 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cdce9d5c7254c08921f71f7cbd1fc7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:29:53.126469 2631 tablet_bootstrap.cc:492] T 1cdce9d5c7254c08921f71f7cbd1fc7b P e59bb6f9de19490a833d59ad8dfae247: Bootstrap starting.
I20260926 11:29:53.128204 2670 ts_tablet_manager.cc:1905] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:29:53.128667 2630 tablet_bootstrap.cc:492] T 1cdce9d5c7254c08921f71f7cbd1fc7b P fe0b582cabc24cb39653fb7f3fbcc549: Bootstrap starting.
I20260926 11:29:53.128679 2400 tablet_service.cc:1467] Processing CreateTablet for tablet 1cdce9d5c7254c08921f71f7cbd1fc7b (DEFAULT_TABLE table=test-table [id=64dc9f7b32b6418392f5680ef802c2c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:29:53.128934 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cdce9d5c7254c08921f71f7cbd1fc7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:29:53.129662 2630 tablet_bootstrap.cc:654] T 1cdce9d5c7254c08921f71f7cbd1fc7b P fe0b582cabc24cb39653fb7f3fbcc549: Neither blocks nor log segments found. Creating new log.
I20260926 11:29:53.131245 2629 tablet_bootstrap.cc:492] T 1cdce9d5c7254c08921f71f7cbd1fc7b P 3af2bda673cf4d7293ad681f9d4e4432: Bootstrap starting.
I20260926 11:29:53.131264 2670 ts_tablet_manager.cc:1918] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 11:29:53.131469 2670 log.cc:1196] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-2/wal/wals/aa94c72f40d54fdbb4d28d1fa782a0e3
I20260926 11:29:53.131552 2631 tablet_bootstrap.cc:654] T 1cdce9d5c7254c08921f71f7cbd1fc7b P e59bb6f9de19490a833d59ad8dfae247: Neither blocks nor log segments found. Creating new log.
I20260926 11:29:53.131923 2670 ts_tablet_manager.cc:1939] T aa94c72f40d54fdbb4d28d1fa782a0e3 P e59bb6f9de19490a833d59ad8dfae247: Deleting consensus metadata
I20260926 11:29:53.132323 2629 tablet_bootstrap.cc:654] T 1cdce9d5c7254c08921f71f7cbd1fc7b P 3af2bda673cf4d7293ad681f9d4e4432: Neither blocks nor log segments found. Creating new log.
I20260926 11:29:53.133316 2630 tablet_bootstrap.cc:492] T 1cdce9d5c7254c08921f71f7cbd1fc7b P fe0b582cabc24cb39653fb7f3fbcc549: No bootstrap required, opened a new log
I20260926 11:29:53.133420 2630 ts_tablet_manager.cc:1397] T 1cdce9d5c7254c08921f71f7cbd1fc7b P fe0b582cabc24cb39653fb7f3fbcc549: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260926 11:29:53.133601 2631 tablet_bootstrap.cc:492] T 1cdce9d5c7254c08921f71f7cbd1fc7b P e59bb6f9de19490a833d59ad8dfae247: No bootstrap required, opened a new log
I20260926 11:29:53.133683 2631 ts_tablet_manager.cc:1397] T 1cdce9d5c7254c08921f71f7cbd1fc7b P e59bb6f9de19490a833d59ad8dfae247: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260926 11:29:53.133874 2629 tablet_bootstrap.cc:492] T 1cdce9d5c7254c08921f71f7cbd1fc7b P 3af2bda673cf4d7293ad681f9d4e4432: No bootstrap required, opened a new log
I20260926 11:29:53.133942 2629 ts_tablet_manager.cc:1397] T 1cdce9d5c7254c08921f71f7cbd1fc7b P 3af2bda673cf4d7293ad681f9d4e4432: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 11:29:53.133850 2630 raft_consensus.cc:348] T 1cdce9d5c7254c08921f71f7cbd1fc7b P fe0b582cabc24cb39653fb7f3fbcc549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:53.133986 2630 raft_consensus.cc:374] T 1cdce9d5c7254c08921f71f7cbd1fc7b P fe0b582cabc24cb39653fb7f3fbcc549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:29:53.134020 2630 raft_consensus.cc:729] T 1cdce9d5c7254c08921f71f7cbd1fc7b P fe0b582cabc24cb39653fb7f3fbcc549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe0b582cabc24cb39653fb7f3fbcc549, State: Initialized, Role: FOLLOWER
I20260926 11:29:53.134148 2630 consensus_queue.cc:260] T 1cdce9d5c7254c08921f71f7cbd1fc7b P fe0b582cabc24cb39653fb7f3fbcc549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:53.134135 2631 raft_consensus.cc:348] T 1cdce9d5c7254c08921f71f7cbd1fc7b P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:53.134238 2631 raft_consensus.cc:374] T 1cdce9d5c7254c08921f71f7cbd1fc7b P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:29:53.134269 2631 raft_consensus.cc:729] T 1cdce9d5c7254c08921f71f7cbd1fc7b P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59bb6f9de19490a833d59ad8dfae247, State: Initialized, Role: FOLLOWER
I20260926 11:29:53.134341 2629 raft_consensus.cc:348] T 1cdce9d5c7254c08921f71f7cbd1fc7b P 3af2bda673cf4d7293ad681f9d4e4432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:53.134411 2630 ts_tablet_manager.cc:1428] T 1cdce9d5c7254c08921f71f7cbd1fc7b P fe0b582cabc24cb39653fb7f3fbcc549: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260926 11:29:53.134388 2631 consensus_queue.cc:260] T 1cdce9d5c7254c08921f71f7cbd1fc7b P e59bb6f9de19490a833d59ad8dfae247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:53.134454 2629 raft_consensus.cc:374] T 1cdce9d5c7254c08921f71f7cbd1fc7b P 3af2bda673cf4d7293ad681f9d4e4432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:29:53.134487 2629 raft_consensus.cc:729] T 1cdce9d5c7254c08921f71f7cbd1fc7b P 3af2bda673cf4d7293ad681f9d4e4432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3af2bda673cf4d7293ad681f9d4e4432, State: Initialized, Role: FOLLOWER
I20260926 11:29:53.134567 2110 catalog_manager.cc:4841] TS e59bb6f9de19490a833d59ad8dfae247 (127.0.98.3:39985): tablet aa94c72f40d54fdbb4d28d1fa782a0e3 (table test-table [id=9f6c0baf7ada448484d1e251047c434c]) successfully deleted
I20260926 11:29:53.134596 2629 consensus_queue.cc:260] T 1cdce9d5c7254c08921f71f7cbd1fc7b P 3af2bda673cf4d7293ad681f9d4e4432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:53.134652 2631 ts_tablet_manager.cc:1428] T 1cdce9d5c7254c08921f71f7cbd1fc7b P e59bb6f9de19490a833d59ad8dfae247: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 11:29:53.134856 2629 ts_tablet_manager.cc:1428] T 1cdce9d5c7254c08921f71f7cbd1fc7b P 3af2bda673cf4d7293ad681f9d4e4432: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 11:29:53.281649 2635 raft_consensus.cc:482] T 1cdce9d5c7254c08921f71f7cbd1fc7b P 3af2bda673cf4d7293ad681f9d4e4432 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:29:53.281857 2635 raft_consensus.cc:504] T 1cdce9d5c7254c08921f71f7cbd1fc7b P 3af2bda673cf4d7293ad681f9d4e4432 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:53.283213 2635 leader_election.cc:288] T 1cdce9d5c7254c08921f71f7cbd1fc7b P 3af2bda673cf4d7293ad681f9d4e4432 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe0b582cabc24cb39653fb7f3fbcc549 (127.0.98.1:40609), e59bb6f9de19490a833d59ad8dfae247 (127.0.98.3:39985)
I20260926 11:29:53.288815 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:53.283619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50097 (local address 127.0.98.1:40609)
0926 11:29:53.283879 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:53.283884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:53.284341 (+ 457us) server_negotiation.cc:407] Connection header received
0926 11:29:53.284559 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:53.284562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:53.284614 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:53.284666 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:29:53.285099 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:53.285868 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:53.287453 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:53.288105 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:53.288186 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:53.288250 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:53.288604 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 11:29:53.288683 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260926 11:29:53.289219 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:53.283435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40609 (local address 127.0.98.2:50097)
0926 11:29:53.284244 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:53.284259 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:53.284428 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:53.284813 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:53.284816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:53.284843 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:29:53.284958 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:53.284965 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:53.286014 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:53.286017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:53.287307 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:53.287318 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:53.288349 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:53.288352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:53.288455 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:53.288479 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 11:29:53.288969 (+ 490us) client_negotiation.cc:240] Negotiation successful
0926 11:29:53.289058 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":1,"thread_start_us":91,"threads_started":1}
I20260926 11:29:53.289721 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cdce9d5c7254c08921f71f7cbd1fc7b" candidate_uuid: "3af2bda673cf4d7293ad681f9d4e4432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" is_pre_election: true
I20260926 11:29:53.289887 2280 raft_consensus.cc:2454] T 1cdce9d5c7254c08921f71f7cbd1fc7b P fe0b582cabc24cb39653fb7f3fbcc549 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3af2bda673cf4d7293ad681f9d4e4432 in term 0.
I20260926 11:29:53.290020 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:53.284055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57161 (local address 127.0.98.3:39985)
0926 11:29:53.284176 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:53.284182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:53.284490 (+ 308us) server_negotiation.cc:407] Connection header received
0926 11:29:53.284712 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:53.284719 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:53.284777 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:53.284836 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:29:53.285325 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:53.286251 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:53.287993 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:53.288796 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:53.288904 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:53.288973 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:53.289747 (+ 774us) server_negotiation.cc:299] Negotiation successful
0926 11:29:53.289850 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 11:29:53.290314 2351 leader_election.cc:302] T 1cdce9d5c7254c08921f71f7cbd1fc7b P 3af2bda673cf4d7293ad681f9d4e4432 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3af2bda673cf4d7293ad681f9d4e4432, fe0b582cabc24cb39653fb7f3fbcc549; no voters:
I20260926 11:29:53.290570 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:53.283917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39985 (local address 127.0.98.2:57161)
0926 11:29:53.284388 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:53.284402 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:53.284572 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:53.285010 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:53.285014 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:53.285037 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:29:53.285178 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:53.285186 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:53.286637 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:53.286642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:53.287816 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:53.287830 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:53.289478 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:53.289481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:53.289577 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:53.289607 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 11:29:53.290374 (+ 767us) client_negotiation.cc:240] Negotiation successful
0926 11:29:53.290448 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":110,"threads_started":1}
I20260926 11:29:53.290599 2635 raft_consensus.cc:2781] T 1cdce9d5c7254c08921f71f7cbd1fc7b P 3af2bda673cf4d7293ad681f9d4e4432 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:29:53.290645 2635 raft_consensus.cc:482] T 1cdce9d5c7254c08921f71f7cbd1fc7b P 3af2bda673cf4d7293ad681f9d4e4432 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:29:53.290690 2635 raft_consensus.cc:3037] T 1cdce9d5c7254c08921f71f7cbd1fc7b P 3af2bda673cf4d7293ad681f9d4e4432 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:29:53.291098 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cdce9d5c7254c08921f71f7cbd1fc7b" candidate_uuid: "3af2bda673cf4d7293ad681f9d4e4432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59bb6f9de19490a833d59ad8dfae247" is_pre_election: true
I20260926 11:29:53.291376 2560 raft_consensus.cc:2454] T 1cdce9d5c7254c08921f71f7cbd1fc7b P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3af2bda673cf4d7293ad681f9d4e4432 in term 0.
I20260926 11:29:53.292285 2635 raft_consensus.cc:504] T 1cdce9d5c7254c08921f71f7cbd1fc7b P 3af2bda673cf4d7293ad681f9d4e4432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:53.292753 2635 leader_election.cc:288] T 1cdce9d5c7254c08921f71f7cbd1fc7b P 3af2bda673cf4d7293ad681f9d4e4432 [CANDIDATE]: Term 1 election: Requested vote from peers fe0b582cabc24cb39653fb7f3fbcc549 (127.0.98.1:40609), e59bb6f9de19490a833d59ad8dfae247 (127.0.98.3:39985)
I20260926 11:29:53.295996 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cdce9d5c7254c08921f71f7cbd1fc7b" candidate_uuid: "3af2bda673cf4d7293ad681f9d4e4432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59bb6f9de19490a833d59ad8dfae247"
I20260926 11:29:53.296118 2560 raft_consensus.cc:3037] T 1cdce9d5c7254c08921f71f7cbd1fc7b P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:29:53.296408 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cdce9d5c7254c08921f71f7cbd1fc7b" candidate_uuid: "3af2bda673cf4d7293ad681f9d4e4432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe0b582cabc24cb39653fb7f3fbcc549"
I20260926 11:29:53.296530 2280 raft_consensus.cc:3037] T 1cdce9d5c7254c08921f71f7cbd1fc7b P fe0b582cabc24cb39653fb7f3fbcc549 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:29:53.297394 2560 raft_consensus.cc:2454] T 1cdce9d5c7254c08921f71f7cbd1fc7b P e59bb6f9de19490a833d59ad8dfae247 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3af2bda673cf4d7293ad681f9d4e4432 in term 1.
I20260926 11:29:53.297744 2354 leader_election.cc:302] T 1cdce9d5c7254c08921f71f7cbd1fc7b P 3af2bda673cf4d7293ad681f9d4e4432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3af2bda673cf4d7293ad681f9d4e4432, e59bb6f9de19490a833d59ad8dfae247; no voters:
I20260926 11:29:53.297777 2280 raft_consensus.cc:2454] T 1cdce9d5c7254c08921f71f7cbd1fc7b P fe0b582cabc24cb39653fb7f3fbcc549 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3af2bda673cf4d7293ad681f9d4e4432 in term 1.
I20260926 11:29:53.297928 2635 raft_consensus.cc:2781] T 1cdce9d5c7254c08921f71f7cbd1fc7b P 3af2bda673cf4d7293ad681f9d4e4432 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:29:53.298128 2635 raft_consensus.cc:686] T 1cdce9d5c7254c08921f71f7cbd1fc7b P 3af2bda673cf4d7293ad681f9d4e4432 [term 1 LEADER]: Becoming Leader. State: Replica: 3af2bda673cf4d7293ad681f9d4e4432, State: Running, Role: LEADER
I20260926 11:29:53.298379 2635 consensus_queue.cc:237] T 1cdce9d5c7254c08921f71f7cbd1fc7b P 3af2bda673cf4d7293ad681f9d4e4432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:53.301313 2122 catalog_manager.cc:5495] T 1cdce9d5c7254c08921f71f7cbd1fc7b P 3af2bda673cf4d7293ad681f9d4e4432 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3af2bda673cf4d7293ad681f9d4e4432 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3af2bda673cf4d7293ad681f9d4e4432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } health_report { overall_health: UNKNOWN } } }
I20260926 11:29:53.339103 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:53.335423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44124 (local address 127.0.98.2:39889)
0926 11:29:53.335559 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:53.335565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:53.335581 (+ 16us) server_negotiation.cc:407] Connection header received
0926 11:29:53.335810 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:53.335814 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:53.335879 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:53.335991 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:29:53.336252 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:53.336769 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:53.337601 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:53.337836 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:53.337947 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:53.338224 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:29:53.338703 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:29:53.338838 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:53.338899 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 11:29:53.338956 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 11:29:53.352142 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:53.347761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44615 (local address 127.0.98.3:39985)
0926 11:29:53.347903 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:53.347908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:53.348090 (+ 182us) server_negotiation.cc:407] Connection header received
0926 11:29:53.348251 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:53.348253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:53.348297 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:53.348362 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:29:53.348740 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:53.349386 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:53.350883 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:53.351100 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:53.351199 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:53.351597 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:29:53.351669 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:29:53.351839 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:53.351922 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 11:29:53.351985 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 11:29:53.352558 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:53.347582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39985 (local address 127.0.98.2:44615)
0926 11:29:53.347961 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:53.347980 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:53.348143 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:53.348397 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:53.348401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:53.348412 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 11:29:53.348579 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:53.348589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:53.349503 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:53.349508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:53.350740 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:53.350755 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:53.351251 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:53.351256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:53.351387 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:53.351456 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 11:29:53.351812 (+ 356us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 11:29:53.351818 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 11:29:53.351909 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 11:29:53.352017 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":146,"threads_started":1}
I20260926 11:29:53.360031 2280 raft_consensus.cc:1261] T 1cdce9d5c7254c08921f71f7cbd1fc7b P fe0b582cabc24cb39653fb7f3fbcc549 [term 1 FOLLOWER]: Refusing update from remote peer 3af2bda673cf4d7293ad681f9d4e4432: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:29:53.360172 2560 raft_consensus.cc:1261] T 1cdce9d5c7254c08921f71f7cbd1fc7b P e59bb6f9de19490a833d59ad8dfae247 [term 1 FOLLOWER]: Refusing update from remote peer 3af2bda673cf4d7293ad681f9d4e4432: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:29:53.360579 2678 consensus_queue.cc:1035] T 1cdce9d5c7254c08921f71f7cbd1fc7b P 3af2bda673cf4d7293ad681f9d4e4432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:29:53.361076 2678 consensus_queue.cc:1035] T 1cdce9d5c7254c08921f71f7cbd1fc7b P 3af2bda673cf4d7293ad681f9d4e4432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:29:53.384241 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:53.379908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39889 (local address 127.0.98.3:36635)
0926 11:29:53.380034 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:53.380051 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:53.380136 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:53.380499 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:53.380505 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:53.380517 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 11:29:53.380642 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:53.380651 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:53.381837 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:53.381840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:53.382614 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:53.382625 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:53.383454 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:53.383459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:53.383581 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:53.383627 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 11:29:53.383997 (+ 370us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 11:29:53.384004 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 11:29:53.384063 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 11:29:53.384133 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 11:29:53.384328 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:53.379977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36635 (local address 127.0.98.2:39889)
0926 11:29:53.380080 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:53.380087 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:53.380102 (+ 15us) server_negotiation.cc:407] Connection header received
0926 11:29:53.380223 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:53.380228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:53.380298 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:53.380370 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:29:53.380771 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:53.381726 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:53.382771 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:53.383239 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:53.383362 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:53.383805 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:29:53.383915 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:29:53.383995 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:53.384079 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 11:29:53.384150 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260926 11:29:53.385018 2656 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 11:29:54.947027 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:29:54.947189 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49060:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:29:54.949600 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 60f3b6fa425a4a4aa98939c454c03905: Sending DeleteTablet for 3 replicas of tablet 1cdce9d5c7254c08921f71f7cbd1fc7b
I20260926 11:29:54.950163 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cdce9d5c7254c08921f71f7cbd1fc7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:29:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53254
I20260926 11:29:54.950271 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cdce9d5c7254c08921f71f7cbd1fc7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:29:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43990
I20260926 11:29:54.950456 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cdce9d5c7254c08921f71f7cbd1fc7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:29:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44118
I20260926 11:29:54.950507 2709 tablet_replica.cc:330] T 1cdce9d5c7254c08921f71f7cbd1fc7b P fe0b582cabc24cb39653fb7f3fbcc549: stopping tablet replica
I20260926 11:29:54.950626 2709 raft_consensus.cc:2229] T 1cdce9d5c7254c08921f71f7cbd1fc7b P fe0b582cabc24cb39653fb7f3fbcc549 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:29:54.950727 2709 raft_consensus.cc:2258] T 1cdce9d5c7254c08921f71f7cbd1fc7b P fe0b582cabc24cb39653fb7f3fbcc549 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:29:54.951463 2709 ts_tablet_manager.cc:1905] T 1cdce9d5c7254c08921f71f7cbd1fc7b P fe0b582cabc24cb39653fb7f3fbcc549: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:29:54.953154 2711 tablet_replica.cc:330] T 1cdce9d5c7254c08921f71f7cbd1fc7b P 3af2bda673cf4d7293ad681f9d4e4432: stopping tablet replica
I20260926 11:29:54.953349 2711 raft_consensus.cc:2229] T 1cdce9d5c7254c08921f71f7cbd1fc7b P 3af2bda673cf4d7293ad681f9d4e4432 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:29:54.954478 2709 ts_tablet_manager.cc:1918] T 1cdce9d5c7254c08921f71f7cbd1fc7b P fe0b582cabc24cb39653fb7f3fbcc549: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 11:29:54.954571 2709 log.cc:1196] T 1cdce9d5c7254c08921f71f7cbd1fc7b P fe0b582cabc24cb39653fb7f3fbcc549: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-0/wal/wals/1cdce9d5c7254c08921f71f7cbd1fc7b
I20260926 11:29:54.954831 2709 ts_tablet_manager.cc:1939] T 1cdce9d5c7254c08921f71f7cbd1fc7b P fe0b582cabc24cb39653fb7f3fbcc549: Deleting consensus metadata
I20260926 11:29:54.955475 2107 catalog_manager.cc:4841] TS fe0b582cabc24cb39653fb7f3fbcc549 (127.0.98.1:40609): tablet 1cdce9d5c7254c08921f71f7cbd1fc7b (table test-table [id=64dc9f7b32b6418392f5680ef802c2c6]) successfully deleted
I20260926 11:29:54.956014 2711 raft_consensus.cc:2258] T 1cdce9d5c7254c08921f71f7cbd1fc7b P 3af2bda673cf4d7293ad681f9d4e4432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:29:54.956307 2710 tablet_replica.cc:330] T 1cdce9d5c7254c08921f71f7cbd1fc7b P e59bb6f9de19490a833d59ad8dfae247: stopping tablet replica
I20260926 11:29:54.956485 2710 raft_consensus.cc:2229] T 1cdce9d5c7254c08921f71f7cbd1fc7b P e59bb6f9de19490a833d59ad8dfae247 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:29:54.956610 2710 raft_consensus.cc:2258] T 1cdce9d5c7254c08921f71f7cbd1fc7b P e59bb6f9de19490a833d59ad8dfae247 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:29:54.956775 2711 ts_tablet_manager.cc:1905] T 1cdce9d5c7254c08921f71f7cbd1fc7b P 3af2bda673cf4d7293ad681f9d4e4432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:29:54.957230 2710 ts_tablet_manager.cc:1905] T 1cdce9d5c7254c08921f71f7cbd1fc7b P e59bb6f9de19490a833d59ad8dfae247: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:29:54.959228 2711 ts_tablet_manager.cc:1918] T 1cdce9d5c7254c08921f71f7cbd1fc7b P 3af2bda673cf4d7293ad681f9d4e4432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 11:29:54.959312 2711 log.cc:1196] T 1cdce9d5c7254c08921f71f7cbd1fc7b P 3af2bda673cf4d7293ad681f9d4e4432: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-1/wal/wals/1cdce9d5c7254c08921f71f7cbd1fc7b
I20260926 11:29:54.959561 2710 ts_tablet_manager.cc:1918] T 1cdce9d5c7254c08921f71f7cbd1fc7b P e59bb6f9de19490a833d59ad8dfae247: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 11:29:54.959621 2710 log.cc:1196] T 1cdce9d5c7254c08921f71f7cbd1fc7b P e59bb6f9de19490a833d59ad8dfae247: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-2/wal/wals/1cdce9d5c7254c08921f71f7cbd1fc7b
I20260926 11:29:54.959616 2711 ts_tablet_manager.cc:1939] T 1cdce9d5c7254c08921f71f7cbd1fc7b P 3af2bda673cf4d7293ad681f9d4e4432: Deleting consensus metadata
I20260926 11:29:54.959892 2710 ts_tablet_manager.cc:1939] T 1cdce9d5c7254c08921f71f7cbd1fc7b P e59bb6f9de19490a833d59ad8dfae247: Deleting consensus metadata
I20260926 11:29:54.960330 2107 catalog_manager.cc:4841] TS 3af2bda673cf4d7293ad681f9d4e4432 (127.0.98.2:39889): tablet 1cdce9d5c7254c08921f71f7cbd1fc7b (table test-table [id=64dc9f7b32b6418392f5680ef802c2c6]) successfully deleted
I20260926 11:29:54.960515 2110 catalog_manager.cc:4841] TS e59bb6f9de19490a833d59ad8dfae247 (127.0.98.3:39985): tablet 1cdce9d5c7254c08921f71f7cbd1fc7b (table test-table [id=64dc9f7b32b6418392f5680ef802c2c6]) successfully deleted
Sep 26 11:29:54 dist-test-slave-mrck krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422194, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 11:29:54 dist-test-slave-mrck krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422194, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:29:54.983964 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:54.970253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49084 (local address 127.0.98.62:33091)
0926 11:29:54.970532 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:54.970540 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:54.970639 (+ 99us) server_negotiation.cc:407] Connection header received
0926 11:29:54.970779 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:54.970782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:54.970834 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:54.970935 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:54.971625 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:54.972243 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:54.973080 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:54.973311 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:54.973433 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:54.980754 (+ 7321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:54.980758 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:54.980760 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:54.980787 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:54.982182 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:54.982621 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:54.982626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:54.982627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:54.982714 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:54.982992 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:54.982996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:54.982998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:54.983185 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:54.983326 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:54.983572 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 11:29:54.983701 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":94,"threads_started":1}
I20260926 11:29:54.986501 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:29:54.986902 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:29:54.991209 2260 tablet_service.cc:1467] Processing CreateTablet for tablet daf25f2995da4e19b548b275838f9e8e (DEFAULT_TABLE table=test-table [id=5eb7e43db9964b298dfffff081a97f81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:29:54.991212 2400 tablet_service.cc:1467] Processing CreateTablet for tablet daf25f2995da4e19b548b275838f9e8e (DEFAULT_TABLE table=test-table [id=5eb7e43db9964b298dfffff081a97f81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:29:54.991304 2539 tablet_service.cc:1467] Processing CreateTablet for tablet daf25f2995da4e19b548b275838f9e8e (DEFAULT_TABLE table=test-table [id=5eb7e43db9964b298dfffff081a97f81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:29:54.991544 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daf25f2995da4e19b548b275838f9e8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:29:54.991551 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daf25f2995da4e19b548b275838f9e8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:29:54.991542 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daf25f2995da4e19b548b275838f9e8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:29:54.994036 2721 tablet_bootstrap.cc:492] T daf25f2995da4e19b548b275838f9e8e P e59bb6f9de19490a833d59ad8dfae247: Bootstrap starting.
I20260926 11:29:54.993635 2720 tablet_bootstrap.cc:492] T daf25f2995da4e19b548b275838f9e8e P fe0b582cabc24cb39653fb7f3fbcc549: Bootstrap starting.
I20260926 11:29:54.994361 2722 tablet_bootstrap.cc:492] T daf25f2995da4e19b548b275838f9e8e P 3af2bda673cf4d7293ad681f9d4e4432: Bootstrap starting.
I20260926 11:29:54.994971 2720 tablet_bootstrap.cc:654] T daf25f2995da4e19b548b275838f9e8e P fe0b582cabc24cb39653fb7f3fbcc549: Neither blocks nor log segments found. Creating new log.
I20260926 11:29:54.994971 2721 tablet_bootstrap.cc:654] T daf25f2995da4e19b548b275838f9e8e P e59bb6f9de19490a833d59ad8dfae247: Neither blocks nor log segments found. Creating new log.
I20260926 11:29:54.995141 2722 tablet_bootstrap.cc:654] T daf25f2995da4e19b548b275838f9e8e P 3af2bda673cf4d7293ad681f9d4e4432: Neither blocks nor log segments found. Creating new log.
I20260926 11:29:54.996156 2721 tablet_bootstrap.cc:492] T daf25f2995da4e19b548b275838f9e8e P e59bb6f9de19490a833d59ad8dfae247: No bootstrap required, opened a new log
I20260926 11:29:54.996173 2720 tablet_bootstrap.cc:492] T daf25f2995da4e19b548b275838f9e8e P fe0b582cabc24cb39653fb7f3fbcc549: No bootstrap required, opened a new log
I20260926 11:29:54.996259 2720 ts_tablet_manager.cc:1397] T daf25f2995da4e19b548b275838f9e8e P fe0b582cabc24cb39653fb7f3fbcc549: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 11:29:54.996265 2721 ts_tablet_manager.cc:1397] T daf25f2995da4e19b548b275838f9e8e P e59bb6f9de19490a833d59ad8dfae247: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 11:29:54.996302 2722 tablet_bootstrap.cc:492] T daf25f2995da4e19b548b275838f9e8e P 3af2bda673cf4d7293ad681f9d4e4432: No bootstrap required, opened a new log
I20260926 11:29:54.996377 2722 ts_tablet_manager.cc:1397] T daf25f2995da4e19b548b275838f9e8e P 3af2bda673cf4d7293ad681f9d4e4432: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 11:29:54.996686 2720 raft_consensus.cc:348] T daf25f2995da4e19b548b275838f9e8e P fe0b582cabc24cb39653fb7f3fbcc549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:54.996712 2721 raft_consensus.cc:348] T daf25f2995da4e19b548b275838f9e8e P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:54.996811 2720 raft_consensus.cc:374] T daf25f2995da4e19b548b275838f9e8e P fe0b582cabc24cb39653fb7f3fbcc549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:29:54.996822 2721 raft_consensus.cc:374] T daf25f2995da4e19b548b275838f9e8e P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:29:54.996845 2720 raft_consensus.cc:729] T daf25f2995da4e19b548b275838f9e8e P fe0b582cabc24cb39653fb7f3fbcc549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe0b582cabc24cb39653fb7f3fbcc549, State: Initialized, Role: FOLLOWER
I20260926 11:29:54.996784 2722 raft_consensus.cc:348] T daf25f2995da4e19b548b275838f9e8e P 3af2bda673cf4d7293ad681f9d4e4432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:54.996858 2721 raft_consensus.cc:729] T daf25f2995da4e19b548b275838f9e8e P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59bb6f9de19490a833d59ad8dfae247, State: Initialized, Role: FOLLOWER
I20260926 11:29:54.996886 2722 raft_consensus.cc:374] T daf25f2995da4e19b548b275838f9e8e P 3af2bda673cf4d7293ad681f9d4e4432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:29:54.996917 2722 raft_consensus.cc:729] T daf25f2995da4e19b548b275838f9e8e P 3af2bda673cf4d7293ad681f9d4e4432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3af2bda673cf4d7293ad681f9d4e4432, State: Initialized, Role: FOLLOWER
I20260926 11:29:54.996994 2721 consensus_queue.cc:260] T daf25f2995da4e19b548b275838f9e8e P e59bb6f9de19490a833d59ad8dfae247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:54.997045 2722 consensus_queue.cc:260] T daf25f2995da4e19b548b275838f9e8e P 3af2bda673cf4d7293ad681f9d4e4432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:54.997279 2721 ts_tablet_manager.cc:1428] T daf25f2995da4e19b548b275838f9e8e P e59bb6f9de19490a833d59ad8dfae247: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 11:29:54.997368 2722 ts_tablet_manager.cc:1428] T daf25f2995da4e19b548b275838f9e8e P 3af2bda673cf4d7293ad681f9d4e4432: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 11:29:54.997036 2720 consensus_queue.cc:260] T daf25f2995da4e19b548b275838f9e8e P fe0b582cabc24cb39653fb7f3fbcc549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:54.997974 2720 ts_tablet_manager.cc:1428] T daf25f2995da4e19b548b275838f9e8e P fe0b582cabc24cb39653fb7f3fbcc549: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 11:29:55.001256 2702 raft_consensus.cc:482] T daf25f2995da4e19b548b275838f9e8e P 3af2bda673cf4d7293ad681f9d4e4432 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:29:55.001334 2702 raft_consensus.cc:504] T daf25f2995da4e19b548b275838f9e8e P 3af2bda673cf4d7293ad681f9d4e4432 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:55.001732 2702 leader_election.cc:288] T daf25f2995da4e19b548b275838f9e8e P 3af2bda673cf4d7293ad681f9d4e4432 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe0b582cabc24cb39653fb7f3fbcc549 (127.0.98.1:40609), e59bb6f9de19490a833d59ad8dfae247 (127.0.98.3:39985)
I20260926 11:29:55.002053 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf25f2995da4e19b548b275838f9e8e" candidate_uuid: "3af2bda673cf4d7293ad681f9d4e4432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" is_pre_election: true
I20260926 11:29:55.002194 2280 raft_consensus.cc:2454] T daf25f2995da4e19b548b275838f9e8e P fe0b582cabc24cb39653fb7f3fbcc549 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3af2bda673cf4d7293ad681f9d4e4432 in term 0.
I20260926 11:29:55.002465 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf25f2995da4e19b548b275838f9e8e" candidate_uuid: "3af2bda673cf4d7293ad681f9d4e4432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59bb6f9de19490a833d59ad8dfae247" is_pre_election: true
I20260926 11:29:55.002496 2351 leader_election.cc:302] T daf25f2995da4e19b548b275838f9e8e P 3af2bda673cf4d7293ad681f9d4e4432 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3af2bda673cf4d7293ad681f9d4e4432, fe0b582cabc24cb39653fb7f3fbcc549; no voters:
I20260926 11:29:55.002595 2560 raft_consensus.cc:2454] T daf25f2995da4e19b548b275838f9e8e P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3af2bda673cf4d7293ad681f9d4e4432 in term 0.
I20260926 11:29:55.002774 2702 raft_consensus.cc:2781] T daf25f2995da4e19b548b275838f9e8e P 3af2bda673cf4d7293ad681f9d4e4432 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:29:55.002817 2702 raft_consensus.cc:482] T daf25f2995da4e19b548b275838f9e8e P 3af2bda673cf4d7293ad681f9d4e4432 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:29:55.002843 2702 raft_consensus.cc:3037] T daf25f2995da4e19b548b275838f9e8e P 3af2bda673cf4d7293ad681f9d4e4432 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:29:55.003830 2702 raft_consensus.cc:504] T daf25f2995da4e19b548b275838f9e8e P 3af2bda673cf4d7293ad681f9d4e4432 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:55.004180 2702 leader_election.cc:288] T daf25f2995da4e19b548b275838f9e8e P 3af2bda673cf4d7293ad681f9d4e4432 [CANDIDATE]: Term 1 election: Requested vote from peers fe0b582cabc24cb39653fb7f3fbcc549 (127.0.98.1:40609), e59bb6f9de19490a833d59ad8dfae247 (127.0.98.3:39985)
I20260926 11:29:55.004456 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf25f2995da4e19b548b275838f9e8e" candidate_uuid: "3af2bda673cf4d7293ad681f9d4e4432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e59bb6f9de19490a833d59ad8dfae247"
I20260926 11:29:55.004465 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "daf25f2995da4e19b548b275838f9e8e" candidate_uuid: "3af2bda673cf4d7293ad681f9d4e4432" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe0b582cabc24cb39653fb7f3fbcc549"
I20260926 11:29:55.004566 2280 raft_consensus.cc:3037] T daf25f2995da4e19b548b275838f9e8e P fe0b582cabc24cb39653fb7f3fbcc549 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:29:55.004566 2560 raft_consensus.cc:3037] T daf25f2995da4e19b548b275838f9e8e P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:29:55.005614 2560 raft_consensus.cc:2454] T daf25f2995da4e19b548b275838f9e8e P e59bb6f9de19490a833d59ad8dfae247 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3af2bda673cf4d7293ad681f9d4e4432 in term 1.
I20260926 11:29:55.005614 2280 raft_consensus.cc:2454] T daf25f2995da4e19b548b275838f9e8e P fe0b582cabc24cb39653fb7f3fbcc549 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3af2bda673cf4d7293ad681f9d4e4432 in term 1.
I20260926 11:29:55.005875 2351 leader_election.cc:302] T daf25f2995da4e19b548b275838f9e8e P 3af2bda673cf4d7293ad681f9d4e4432 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3af2bda673cf4d7293ad681f9d4e4432, fe0b582cabc24cb39653fb7f3fbcc549; no voters:
I20260926 11:29:55.006052 2702 raft_consensus.cc:2781] T daf25f2995da4e19b548b275838f9e8e P 3af2bda673cf4d7293ad681f9d4e4432 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:29:55.006139 2702 raft_consensus.cc:686] T daf25f2995da4e19b548b275838f9e8e P 3af2bda673cf4d7293ad681f9d4e4432 [term 1 LEADER]: Becoming Leader. State: Replica: 3af2bda673cf4d7293ad681f9d4e4432, State: Running, Role: LEADER
I20260926 11:29:55.006270 2702 consensus_queue.cc:237] T daf25f2995da4e19b548b275838f9e8e P 3af2bda673cf4d7293ad681f9d4e4432 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:55.007716 2123 catalog_manager.cc:5495] T daf25f2995da4e19b548b275838f9e8e P 3af2bda673cf4d7293ad681f9d4e4432 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3af2bda673cf4d7293ad681f9d4e4432 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "3af2bda673cf4d7293ad681f9d4e4432" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } health_report { overall_health: UNKNOWN } } }
I20260926 11:29:55.026922 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:55.023800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44138 (local address 127.0.98.2:39889)
0926 11:29:55.024052 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:55.024065 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:55.024076 (+ 11us) server_negotiation.cc:407] Connection header received
0926 11:29:55.024112 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:55.024114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:55.024151 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:55.024218 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:29:55.024422 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:55.024928 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:55.025734 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:55.025980 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:55.026058 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:55.026252 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:29:55.026315 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:29:55.026383 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:55.026639 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 11:29:55.026750 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":126,"threads_started":1}
I20260926 11:29:55.032796 2280 raft_consensus.cc:1261] T daf25f2995da4e19b548b275838f9e8e P fe0b582cabc24cb39653fb7f3fbcc549 [term 1 FOLLOWER]: Refusing update from remote peer 3af2bda673cf4d7293ad681f9d4e4432: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:29:55.032796 2560 raft_consensus.cc:1261] T daf25f2995da4e19b548b275838f9e8e P e59bb6f9de19490a833d59ad8dfae247 [term 1 FOLLOWER]: Refusing update from remote peer 3af2bda673cf4d7293ad681f9d4e4432: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:29:55.033187 2702 consensus_queue.cc:1035] T daf25f2995da4e19b548b275838f9e8e P 3af2bda673cf4d7293ad681f9d4e4432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:29:55.033367 2687 consensus_queue.cc:1035] T daf25f2995da4e19b548b275838f9e8e P 3af2bda673cf4d7293ad681f9d4e4432 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:29:55.063162 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:29:55.063313 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:29:55.065241 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 60f3b6fa425a4a4aa98939c454c03905: Sending DeleteTablet for 3 replicas of tablet daf25f2995da4e19b548b275838f9e8e
I20260926 11:29:55.065902 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet daf25f2995da4e19b548b275838f9e8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:29:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44118
I20260926 11:29:55.065922 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet daf25f2995da4e19b548b275838f9e8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:29:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43990
I20260926 11:29:55.065908 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet daf25f2995da4e19b548b275838f9e8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:29:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53254
I20260926 11:29:55.066100 2710 tablet_replica.cc:330] T daf25f2995da4e19b548b275838f9e8e P e59bb6f9de19490a833d59ad8dfae247: stopping tablet replica
I20260926 11:29:55.066105 2711 tablet_replica.cc:330] T daf25f2995da4e19b548b275838f9e8e P 3af2bda673cf4d7293ad681f9d4e4432: stopping tablet replica
I20260926 11:29:55.066210 2711 raft_consensus.cc:2229] T daf25f2995da4e19b548b275838f9e8e P 3af2bda673cf4d7293ad681f9d4e4432 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:29:55.066210 2710 raft_consensus.cc:2229] T daf25f2995da4e19b548b275838f9e8e P e59bb6f9de19490a833d59ad8dfae247 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:29:55.066318 2710 raft_consensus.cc:2258] T daf25f2995da4e19b548b275838f9e8e P e59bb6f9de19490a833d59ad8dfae247 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:29:55.066395 2711 raft_consensus.cc:2258] T daf25f2995da4e19b548b275838f9e8e P 3af2bda673cf4d7293ad681f9d4e4432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:29:55.066516 2709 tablet_replica.cc:330] T daf25f2995da4e19b548b275838f9e8e P fe0b582cabc24cb39653fb7f3fbcc549: stopping tablet replica
I20260926 11:29:55.066658 2709 raft_consensus.cc:2229] T daf25f2995da4e19b548b275838f9e8e P fe0b582cabc24cb39653fb7f3fbcc549 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:29:55.066797 2709 raft_consensus.cc:2258] T daf25f2995da4e19b548b275838f9e8e P fe0b582cabc24cb39653fb7f3fbcc549 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:29:55.067173 2711 ts_tablet_manager.cc:1905] T daf25f2995da4e19b548b275838f9e8e P 3af2bda673cf4d7293ad681f9d4e4432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:29:55.067087 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 11:29:55.067183 2710 ts_tablet_manager.cc:1905] T daf25f2995da4e19b548b275838f9e8e P e59bb6f9de19490a833d59ad8dfae247: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 11:29:55.067492 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:29:55.069229 2709 ts_tablet_manager.cc:1905] T daf25f2995da4e19b548b275838f9e8e P fe0b582cabc24cb39653fb7f3fbcc549: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:29:55.069875 2711 ts_tablet_manager.cc:1918] T daf25f2995da4e19b548b275838f9e8e P 3af2bda673cf4d7293ad681f9d4e4432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 11:29:55.069888 2710 ts_tablet_manager.cc:1918] T daf25f2995da4e19b548b275838f9e8e P e59bb6f9de19490a833d59ad8dfae247: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 11:29:55.069950 2711 log.cc:1196] T daf25f2995da4e19b548b275838f9e8e P 3af2bda673cf4d7293ad681f9d4e4432: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-1/wal/wals/daf25f2995da4e19b548b275838f9e8e
I20260926 11:29:55.069980 2710 log.cc:1196] T daf25f2995da4e19b548b275838f9e8e P e59bb6f9de19490a833d59ad8dfae247: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-2/wal/wals/daf25f2995da4e19b548b275838f9e8e
I20260926 11:29:55.070214 2711 ts_tablet_manager.cc:1939] T daf25f2995da4e19b548b275838f9e8e P 3af2bda673cf4d7293ad681f9d4e4432: Deleting consensus metadata
I20260926 11:29:55.070250 2710 ts_tablet_manager.cc:1939] T daf25f2995da4e19b548b275838f9e8e P e59bb6f9de19490a833d59ad8dfae247: Deleting consensus metadata
I20260926 11:29:55.070914 2107 catalog_manager.cc:4841] TS 3af2bda673cf4d7293ad681f9d4e4432 (127.0.98.2:39889): tablet daf25f2995da4e19b548b275838f9e8e (table test-table [id=5eb7e43db9964b298dfffff081a97f81]) successfully deleted
I20260926 11:29:55.071955 2709 ts_tablet_manager.cc:1918] T daf25f2995da4e19b548b275838f9e8e P fe0b582cabc24cb39653fb7f3fbcc549: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 11:29:55.072053 2709 log.cc:1196] T daf25f2995da4e19b548b275838f9e8e P fe0b582cabc24cb39653fb7f3fbcc549: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-0/wal/wals/daf25f2995da4e19b548b275838f9e8e
I20260926 11:29:55.072317 2709 ts_tablet_manager.cc:1939] T daf25f2995da4e19b548b275838f9e8e P fe0b582cabc24cb39653fb7f3fbcc549: Deleting consensus metadata
I20260926 11:29:55.072386 2110 catalog_manager.cc:4841] TS e59bb6f9de19490a833d59ad8dfae247 (127.0.98.3:39985): tablet daf25f2995da4e19b548b275838f9e8e (table test-table [id=5eb7e43db9964b298dfffff081a97f81]) successfully deleted
I20260926 11:29:55.072962 2107 catalog_manager.cc:4841] TS fe0b582cabc24cb39653fb7f3fbcc549 (127.0.98.1:40609): tablet daf25f2995da4e19b548b275838f9e8e (table test-table [id=5eb7e43db9964b298dfffff081a97f81]) successfully deleted
I20260926 11:29:55.073678 2400 tablet_service.cc:1467] Processing CreateTablet for tablet 6d35c8f75df84220ace04ab3184731ea (DEFAULT_TABLE table=test-table [id=94d13c6b9745464ab620442844814b03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:29:55.073709 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 6d35c8f75df84220ace04ab3184731ea (DEFAULT_TABLE table=test-table [id=94d13c6b9745464ab620442844814b03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:29:55.073951 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d35c8f75df84220ace04ab3184731ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:29:55.073964 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d35c8f75df84220ace04ab3184731ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:29:55.073967 2539 tablet_service.cc:1467] Processing CreateTablet for tablet 6d35c8f75df84220ace04ab3184731ea (DEFAULT_TABLE table=test-table [id=94d13c6b9745464ab620442844814b03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:29:55.074199 2539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d35c8f75df84220ace04ab3184731ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:29:55.075985 2720 tablet_bootstrap.cc:492] T 6d35c8f75df84220ace04ab3184731ea P fe0b582cabc24cb39653fb7f3fbcc549: Bootstrap starting.
I20260926 11:29:55.076284 2722 tablet_bootstrap.cc:492] T 6d35c8f75df84220ace04ab3184731ea P 3af2bda673cf4d7293ad681f9d4e4432: Bootstrap starting.
I20260926 11:29:55.076566 2721 tablet_bootstrap.cc:492] T 6d35c8f75df84220ace04ab3184731ea P e59bb6f9de19490a833d59ad8dfae247: Bootstrap starting.
I20260926 11:29:55.077003 2720 tablet_bootstrap.cc:654] T 6d35c8f75df84220ace04ab3184731ea P fe0b582cabc24cb39653fb7f3fbcc549: Neither blocks nor log segments found. Creating new log.
I20260926 11:29:55.077072 2722 tablet_bootstrap.cc:654] T 6d35c8f75df84220ace04ab3184731ea P 3af2bda673cf4d7293ad681f9d4e4432: Neither blocks nor log segments found. Creating new log.
I20260926 11:29:55.077389 2721 tablet_bootstrap.cc:654] T 6d35c8f75df84220ace04ab3184731ea P e59bb6f9de19490a833d59ad8dfae247: Neither blocks nor log segments found. Creating new log.
I20260926 11:29:55.078117 2720 tablet_bootstrap.cc:492] T 6d35c8f75df84220ace04ab3184731ea P fe0b582cabc24cb39653fb7f3fbcc549: No bootstrap required, opened a new log
I20260926 11:29:55.078117 2722 tablet_bootstrap.cc:492] T 6d35c8f75df84220ace04ab3184731ea P 3af2bda673cf4d7293ad681f9d4e4432: No bootstrap required, opened a new log
I20260926 11:29:55.078203 2720 ts_tablet_manager.cc:1397] T 6d35c8f75df84220ace04ab3184731ea P fe0b582cabc24cb39653fb7f3fbcc549: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 11:29:55.078203 2722 ts_tablet_manager.cc:1397] T 6d35c8f75df84220ace04ab3184731ea P 3af2bda673cf4d7293ad681f9d4e4432: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 11:29:55.078617 2720 raft_consensus.cc:348] T 6d35c8f75df84220ace04ab3184731ea P fe0b582cabc24cb39653fb7f3fbcc549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:55.078617 2722 raft_consensus.cc:348] T 6d35c8f75df84220ace04ab3184731ea P 3af2bda673cf4d7293ad681f9d4e4432 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:55.078737 2720 raft_consensus.cc:374] T 6d35c8f75df84220ace04ab3184731ea P fe0b582cabc24cb39653fb7f3fbcc549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:29:55.078737 2722 raft_consensus.cc:374] T 6d35c8f75df84220ace04ab3184731ea P 3af2bda673cf4d7293ad681f9d4e4432 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:29:55.078778 2722 raft_consensus.cc:729] T 6d35c8f75df84220ace04ab3184731ea P 3af2bda673cf4d7293ad681f9d4e4432 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3af2bda673cf4d7293ad681f9d4e4432, State: Initialized, Role: FOLLOWER
I20260926 11:29:55.078776 2720 raft_consensus.cc:729] T 6d35c8f75df84220ace04ab3184731ea P fe0b582cabc24cb39653fb7f3fbcc549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe0b582cabc24cb39653fb7f3fbcc549, State: Initialized, Role: FOLLOWER
I20260926 11:29:55.078899 2722 consensus_queue.cc:260] T 6d35c8f75df84220ace04ab3184731ea P 3af2bda673cf4d7293ad681f9d4e4432 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:55.078910 2720 consensus_queue.cc:260] T 6d35c8f75df84220ace04ab3184731ea P fe0b582cabc24cb39653fb7f3fbcc549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:55.079170 2722 ts_tablet_manager.cc:1428] T 6d35c8f75df84220ace04ab3184731ea P 3af2bda673cf4d7293ad681f9d4e4432: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 11:29:55.079170 2720 ts_tablet_manager.cc:1428] T 6d35c8f75df84220ace04ab3184731ea P fe0b582cabc24cb39653fb7f3fbcc549: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 11:29:55.079964 2721 tablet_bootstrap.cc:492] T 6d35c8f75df84220ace04ab3184731ea P e59bb6f9de19490a833d59ad8dfae247: No bootstrap required, opened a new log
I20260926 11:29:55.080035 2721 ts_tablet_manager.cc:1397] T 6d35c8f75df84220ace04ab3184731ea P e59bb6f9de19490a833d59ad8dfae247: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 11:29:55.080338 2721 raft_consensus.cc:348] T 6d35c8f75df84220ace04ab3184731ea P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:55.080430 2721 raft_consensus.cc:374] T 6d35c8f75df84220ace04ab3184731ea P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:29:55.080483 2721 raft_consensus.cc:729] T 6d35c8f75df84220ace04ab3184731ea P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e59bb6f9de19490a833d59ad8dfae247, State: Initialized, Role: FOLLOWER
I20260926 11:29:55.080608 2721 consensus_queue.cc:260] T 6d35c8f75df84220ace04ab3184731ea P e59bb6f9de19490a833d59ad8dfae247 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:55.080907 2721 ts_tablet_manager.cc:1428] T 6d35c8f75df84220ace04ab3184731ea P e59bb6f9de19490a833d59ad8dfae247: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 11:29:55.231436 2704 raft_consensus.cc:482] T 6d35c8f75df84220ace04ab3184731ea P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:29:55.231577 2704 raft_consensus.cc:504] T 6d35c8f75df84220ace04ab3184731ea P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:55.232025 2704 leader_election.cc:288] T 6d35c8f75df84220ace04ab3184731ea P e59bb6f9de19490a833d59ad8dfae247 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe0b582cabc24cb39653fb7f3fbcc549 (127.0.98.1:40609), 3af2bda673cf4d7293ad681f9d4e4432 (127.0.98.2:39889)
I20260926 11:29:55.232445 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d35c8f75df84220ace04ab3184731ea" candidate_uuid: "e59bb6f9de19490a833d59ad8dfae247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" is_pre_election: true
I20260926 11:29:55.232560 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d35c8f75df84220ace04ab3184731ea" candidate_uuid: "e59bb6f9de19490a833d59ad8dfae247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3af2bda673cf4d7293ad681f9d4e4432" is_pre_election: true
I20260926 11:29:55.232618 2280 raft_consensus.cc:2454] T 6d35c8f75df84220ace04ab3184731ea P fe0b582cabc24cb39653fb7f3fbcc549 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59bb6f9de19490a833d59ad8dfae247 in term 0.
I20260926 11:29:55.232700 2420 raft_consensus.cc:2454] T 6d35c8f75df84220ace04ab3184731ea P 3af2bda673cf4d7293ad681f9d4e4432 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e59bb6f9de19490a833d59ad8dfae247 in term 0.
I20260926 11:29:55.232950 2491 leader_election.cc:302] T 6d35c8f75df84220ace04ab3184731ea P e59bb6f9de19490a833d59ad8dfae247 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e59bb6f9de19490a833d59ad8dfae247, fe0b582cabc24cb39653fb7f3fbcc549; no voters:
I20260926 11:29:55.233162 2704 raft_consensus.cc:2781] T 6d35c8f75df84220ace04ab3184731ea P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:29:55.233225 2704 raft_consensus.cc:482] T 6d35c8f75df84220ace04ab3184731ea P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:29:55.233265 2704 raft_consensus.cc:3037] T 6d35c8f75df84220ace04ab3184731ea P e59bb6f9de19490a833d59ad8dfae247 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:29:55.234274 2704 raft_consensus.cc:504] T 6d35c8f75df84220ace04ab3184731ea P e59bb6f9de19490a833d59ad8dfae247 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:55.234627 2704 leader_election.cc:288] T 6d35c8f75df84220ace04ab3184731ea P e59bb6f9de19490a833d59ad8dfae247 [CANDIDATE]: Term 1 election: Requested vote from peers fe0b582cabc24cb39653fb7f3fbcc549 (127.0.98.1:40609), 3af2bda673cf4d7293ad681f9d4e4432 (127.0.98.2:39889)
I20260926 11:29:55.234963 2280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d35c8f75df84220ace04ab3184731ea" candidate_uuid: "e59bb6f9de19490a833d59ad8dfae247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe0b582cabc24cb39653fb7f3fbcc549"
I20260926 11:29:55.234983 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d35c8f75df84220ace04ab3184731ea" candidate_uuid: "e59bb6f9de19490a833d59ad8dfae247" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3af2bda673cf4d7293ad681f9d4e4432"
I20260926 11:29:55.235069 2280 raft_consensus.cc:3037] T 6d35c8f75df84220ace04ab3184731ea P fe0b582cabc24cb39653fb7f3fbcc549 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:29:55.235107 2420 raft_consensus.cc:3037] T 6d35c8f75df84220ace04ab3184731ea P 3af2bda673cf4d7293ad681f9d4e4432 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:29:55.236073 2420 raft_consensus.cc:2454] T 6d35c8f75df84220ace04ab3184731ea P 3af2bda673cf4d7293ad681f9d4e4432 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59bb6f9de19490a833d59ad8dfae247 in term 1.
I20260926 11:29:55.236156 2280 raft_consensus.cc:2454] T 6d35c8f75df84220ace04ab3184731ea P fe0b582cabc24cb39653fb7f3fbcc549 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e59bb6f9de19490a833d59ad8dfae247 in term 1.
I20260926 11:29:55.236349 2491 leader_election.cc:302] T 6d35c8f75df84220ace04ab3184731ea P e59bb6f9de19490a833d59ad8dfae247 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3af2bda673cf4d7293ad681f9d4e4432, e59bb6f9de19490a833d59ad8dfae247; no voters:
I20260926 11:29:55.236526 2704 raft_consensus.cc:2781] T 6d35c8f75df84220ace04ab3184731ea P e59bb6f9de19490a833d59ad8dfae247 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:29:55.236632 2704 raft_consensus.cc:686] T 6d35c8f75df84220ace04ab3184731ea P e59bb6f9de19490a833d59ad8dfae247 [term 1 LEADER]: Becoming Leader. State: Replica: e59bb6f9de19490a833d59ad8dfae247, State: Running, Role: LEADER
I20260926 11:29:55.236774 2704 consensus_queue.cc:237] T 6d35c8f75df84220ace04ab3184731ea P e59bb6f9de19490a833d59ad8dfae247 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } }
I20260926 11:29:55.238441 2123 catalog_manager.cc:5495] T 6d35c8f75df84220ace04ab3184731ea P e59bb6f9de19490a833d59ad8dfae247 reported cstate change: term changed from 0 to 1, leader changed from <none> to e59bb6f9de19490a833d59ad8dfae247 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e59bb6f9de19490a833d59ad8dfae247" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e59bb6f9de19490a833d59ad8dfae247" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39985 } health_report { overall_health: HEALTHY } } }
I20260926 11:29:55.275993 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:55.272831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44016 (local address 127.0.98.3:39985)
0926 11:29:55.273160 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:55.273172 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:55.273186 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:29:55.273227 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:55.273229 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:55.273273 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:55.273354 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:29:55.273599 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:55.274198 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:55.275045 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:55.275264 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:55.275351 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:55.275614 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:29:55.275684 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:29:55.275761 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:55.275823 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 11:29:55.275874 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":160,"threads_started":1}
I20260926 11:29:55.281741 2280 raft_consensus.cc:1261] T 6d35c8f75df84220ace04ab3184731ea P fe0b582cabc24cb39653fb7f3fbcc549 [term 1 FOLLOWER]: Refusing update from remote peer e59bb6f9de19490a833d59ad8dfae247: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:29:55.281807 2420 raft_consensus.cc:1261] T 6d35c8f75df84220ace04ab3184731ea P 3af2bda673cf4d7293ad681f9d4e4432 [term 1 FOLLOWER]: Refusing update from remote peer e59bb6f9de19490a833d59ad8dfae247: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:29:55.282135 2704 consensus_queue.cc:1035] T 6d35c8f75df84220ace04ab3184731ea P e59bb6f9de19490a833d59ad8dfae247 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3af2bda673cf4d7293ad681f9d4e4432" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:29:55.282256 2690 consensus_queue.cc:1035] T 6d35c8f75df84220ace04ab3184731ea P e59bb6f9de19490a833d59ad8dfae247 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe0b582cabc24cb39653fb7f3fbcc549" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 11:29:55.295408 2688 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 11:29:56.841349 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:29:56.841527 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:29:56.843807 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 60f3b6fa425a4a4aa98939c454c03905: Sending DeleteTablet for 3 replicas of tablet 6d35c8f75df84220ace04ab3184731ea
I20260926 11:29:56.844301 2260 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d35c8f75df84220ace04ab3184731ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:29:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53254
I20260926 11:29:56.844340 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d35c8f75df84220ace04ab3184731ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:29:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44118
I20260926 11:29:56.844497 2539 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d35c8f75df84220ace04ab3184731ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:29:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43990
I20260926 11:29:56.844667 2763 tablet_replica.cc:330] T 6d35c8f75df84220ace04ab3184731ea P fe0b582cabc24cb39653fb7f3fbcc549: stopping tablet replica
I20260926 11:29:56.844785 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 2195
I20260926 11:29:56.844802 2764 tablet_replica.cc:330] T 6d35c8f75df84220ace04ab3184731ea P e59bb6f9de19490a833d59ad8dfae247: stopping tablet replica
I20260926 11:29:56.844838 2763 raft_consensus.cc:2229] T 6d35c8f75df84220ace04ab3184731ea P fe0b582cabc24cb39653fb7f3fbcc549 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:29:56.845063 2764 raft_consensus.cc:2229] T 6d35c8f75df84220ace04ab3184731ea P e59bb6f9de19490a833d59ad8dfae247 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:29:56.845206 2764 raft_consensus.cc:2258] T 6d35c8f75df84220ace04ab3184731ea P e59bb6f9de19490a833d59ad8dfae247 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:29:56.845479 2762 tablet_replica.cc:330] T 6d35c8f75df84220ace04ab3184731ea P 3af2bda673cf4d7293ad681f9d4e4432: stopping tablet replica
I20260926 11:29:56.845562 2762 raft_consensus.cc:2229] T 6d35c8f75df84220ace04ab3184731ea P 3af2bda673cf4d7293ad681f9d4e4432 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:29:56.845633 2762 raft_consensus.cc:2258] T 6d35c8f75df84220ace04ab3184731ea P 3af2bda673cf4d7293ad681f9d4e4432 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:29:56.846145 2762 ts_tablet_manager.cc:1905] T 6d35c8f75df84220ace04ab3184731ea P 3af2bda673cf4d7293ad681f9d4e4432: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:29:56.847085 2764 ts_tablet_manager.cc:1905] T 6d35c8f75df84220ace04ab3184731ea P e59bb6f9de19490a833d59ad8dfae247: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:29:56.848794 2762 ts_tablet_manager.cc:1918] T 6d35c8f75df84220ace04ab3184731ea P 3af2bda673cf4d7293ad681f9d4e4432: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 11:29:56.848913 2762 log.cc:1196] T 6d35c8f75df84220ace04ab3184731ea P 3af2bda673cf4d7293ad681f9d4e4432: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-1/wal/wals/6d35c8f75df84220ace04ab3184731ea
I20260926 11:29:56.849210 2762 ts_tablet_manager.cc:1939] T 6d35c8f75df84220ace04ab3184731ea P 3af2bda673cf4d7293ad681f9d4e4432: Deleting consensus metadata
I20260926 11:29:56.849892 2764 ts_tablet_manager.cc:1918] T 6d35c8f75df84220ace04ab3184731ea P e59bb6f9de19490a833d59ad8dfae247: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 11:29:56.849933 2107 catalog_manager.cc:4841] TS 3af2bda673cf4d7293ad681f9d4e4432 (127.0.98.2:39889): tablet 6d35c8f75df84220ace04ab3184731ea (table test-table [id=94d13c6b9745464ab620442844814b03]) successfully deleted
I20260926 11:29:56.849984 2764 log.cc:1196] T 6d35c8f75df84220ace04ab3184731ea P e59bb6f9de19490a833d59ad8dfae247: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790422180210734-392-0/minicluster-data/ts-2/wal/wals/6d35c8f75df84220ace04ab3184731ea
I20260926 11:29:56.850246 2764 ts_tablet_manager.cc:1939] T 6d35c8f75df84220ace04ab3184731ea P e59bb6f9de19490a833d59ad8dfae247: Deleting consensus metadata
I20260926 11:29:56.850852 2110 catalog_manager.cc:4841] TS e59bb6f9de19490a833d59ad8dfae247 (127.0.98.3:39985): tablet 6d35c8f75df84220ace04ab3184731ea (table test-table [id=94d13c6b9745464ab620442844814b03]) successfully deleted
W20260926 11:29:56.852571 2107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40609 (error 108)
I20260926 11:29:56.852883 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 2335
I20260926 11:29:56.854089 2766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:56.853702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40609 (local address 127.0.0.1:53258)
0926 11:29:56.853938 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:56.854011 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":95,"threads_started":1}
W20260926 11:29:56.854323 2107 catalog_manager.cc:4584] TS fe0b582cabc24cb39653fb7f3fbcc549 (127.0.98.1:40609): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d35c8f75df84220ace04ab3184731ea: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40609: connect: Connection refused (error 111)
I20260926 11:29:56.860330 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 2475
I20260926 11:29:56.868145 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 2091
2026-09-26T11:29:56Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6278 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:29:56 dist-test-slave-mrck krb5kdc[2769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:29:56 dist-test-slave-mrck krb5kdc[2769](info): set up 2 sockets
Sep 26 11:29:56 dist-test-slave-mrck krb5kdc[2769](info): commencing operation
krb5kdc: starting...
W20260926 11:29:57.891700 392 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:29:57 dist-test-slave-mrck krb5kdc[2769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:29:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:29:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:29:58.008826 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41391
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41391
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:29:58.117812 2782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:29:58.118088 2782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:29:58.118129 2782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:29:58.121929 2782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:29:58.121980 2782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:29:58.121999 2782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:29:58.122015 2782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:29:58.122030 2782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:29:58.127158 2782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:29:58.128188 2782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:29:58.129112 2782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:29:58.137761 2790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:29:58.137915 2788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:29:58.137861 2782 server_base.cc:1034] running on GCE node
W20260926 11:29:58.137771 2787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:29:58.138859 2782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:29:58.140049 2782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:29:58.141235 2782 hybrid_clock.cc:648] HybridClock initialized: now 1790422198141202 us; error 48 us; skew 500 ppm
Sep 26 11:29:58 dist-test-slave-mrck krb5kdc[2769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:29:58.144109 2782 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:29:58.144958 2782 webserver.cc:460] Webserver started at http://127.0.98.62:46285/ using document root <none> and password file <none>
I20260926 11:29:58.145545 2782 fs_manager.cc:359] Metadata directory not provided
I20260926 11:29:58.145607 2782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:29:58.145841 2782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:29:58.147893 2782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "4d8260d18ea34ac8a0c969296510df12"
format_stamp: "Formatted at 2026-09-26 11:29:58 on dist-test-slave-mrck"
server_key: "dd01585adeb41b038c19c8e1d47c37f4"
server_key_iv: "5016a2d5dd8b90ded0115e75069672fb"
server_key_version: "encryptionkey@0"
I20260926 11:29:58.148463 2782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "4d8260d18ea34ac8a0c969296510df12"
format_stamp: "Formatted at 2026-09-26 11:29:58 on dist-test-slave-mrck"
server_key: "dd01585adeb41b038c19c8e1d47c37f4"
server_key_iv: "5016a2d5dd8b90ded0115e75069672fb"
server_key_version: "encryptionkey@0"
I20260926 11:29:58.152096 2782 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:29:58.154330 2797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:58.155514 2782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:29:58.155634 2782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "4d8260d18ea34ac8a0c969296510df12"
format_stamp: "Formatted at 2026-09-26 11:29:58 on dist-test-slave-mrck"
server_key: "dd01585adeb41b038c19c8e1d47c37f4"
server_key_iv: "5016a2d5dd8b90ded0115e75069672fb"
server_key_version: "encryptionkey@0"
I20260926 11:29:58.155771 2782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:29:58.167280 2782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:29:58.170099 2782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:29:58.170287 2782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:29:58.177253 2782 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41391
I20260926 11:29:58.177265 2849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41391 every 8 connection(s)
I20260926 11:29:58.178205 2782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:29:58.180938 2850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:29:58.184561 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 2782
I20260926 11:29:58.184676 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:29:58.184968 392 external_mini_cluster.cc:1428] Setting key f72b7270f49e3129a633e2cbfe561dde
I20260926 11:29:58.186739 2850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d8260d18ea34ac8a0c969296510df12: Bootstrap starting.
I20260926 11:29:58.189257 2850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d8260d18ea34ac8a0c969296510df12: Neither blocks nor log segments found. Creating new log.
I20260926 11:29:58.189988 2850 log.cc:824] T 00000000000000000000000000000000 P 4d8260d18ea34ac8a0c969296510df12: Log is configured to *not* fsync() on all Append() calls
I20260926 11:29:58.192078 2850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d8260d18ea34ac8a0c969296510df12: No bootstrap required, opened a new log
Sep 26 11:29:58 dist-test-slave-mrck krb5kdc[2769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:29:58.195531 2850 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d8260d18ea34ac8a0c969296510df12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d8260d18ea34ac8a0c969296510df12" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41391 } }
I20260926 11:29:58.195837 2850 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d8260d18ea34ac8a0c969296510df12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:29:58.195931 2850 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d8260d18ea34ac8a0c969296510df12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d8260d18ea34ac8a0c969296510df12, State: Initialized, Role: FOLLOWER
I20260926 11:29:58.196382 2850 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d8260d18ea34ac8a0c969296510df12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d8260d18ea34ac8a0c969296510df12" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41391 } }
I20260926 11:29:58.196532 2850 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d8260d18ea34ac8a0c969296510df12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:29:58.196619 2850 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d8260d18ea34ac8a0c969296510df12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:29:58.196733 2850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d8260d18ea34ac8a0c969296510df12 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:29:58.197582 2850 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d8260d18ea34ac8a0c969296510df12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d8260d18ea34ac8a0c969296510df12" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41391 } }
I20260926 11:29:58.197914 2850 leader_election.cc:302] T 00000000000000000000000000000000 P 4d8260d18ea34ac8a0c969296510df12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d8260d18ea34ac8a0c969296510df12; no voters:
I20260926 11:29:58.198237 2850 leader_election.cc:288] T 00000000000000000000000000000000 P 4d8260d18ea34ac8a0c969296510df12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:29:58.198304 2855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d8260d18ea34ac8a0c969296510df12 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:29:58.198549 2855 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d8260d18ea34ac8a0c969296510df12 [term 1 LEADER]: Becoming Leader. State: Replica: 4d8260d18ea34ac8a0c969296510df12, State: Running, Role: LEADER
I20260926 11:29:58.198892 2855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d8260d18ea34ac8a0c969296510df12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d8260d18ea34ac8a0c969296510df12" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41391 } }
I20260926 11:29:58.199564 2850 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d8260d18ea34ac8a0c969296510df12 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:29:58.200131 2853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:58.186301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60668 (local address 127.0.98.62:41391)
0926 11:29:58.186714 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:58.186730 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:58.186764 (+ 34us) server_negotiation.cc:407] Connection header received
0926 11:29:58.187432 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:58.187448 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:58.187827 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:58.188128 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:58.188990 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:58.190023 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:58.191109 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:58.191571 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:58.191686 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:58.194416 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:58.194421 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:58.194449 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:58.194474 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:58.196797 (+ 2323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:58.197236 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:58.197242 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:58.197247 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:58.197362 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:58.197652 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:58.197657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:58.197658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:58.198101 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:58.198306 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:58.198922 (+ 616us) server_negotiation.cc:299] Negotiation successful
0926 11:29:58.199209 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":142,"threads_started":1}
I20260926 11:29:58.200567 2857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d8260d18ea34ac8a0c969296510df12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d8260d18ea34ac8a0c969296510df12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d8260d18ea34ac8a0c969296510df12" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41391 } } }
I20260926 11:29:58.200686 2857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d8260d18ea34ac8a0c969296510df12 [sys.catalog]: This master's current role is: LEADER
I20260926 11:29:58.200618 2856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d8260d18ea34ac8a0c969296510df12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d8260d18ea34ac8a0c969296510df12. Latest consensus state: current_term: 1 leader_uuid: "4d8260d18ea34ac8a0c969296510df12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d8260d18ea34ac8a0c969296510df12" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41391 } } }
I20260926 11:29:58.200750 2856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d8260d18ea34ac8a0c969296510df12 [sys.catalog]: This master's current role is: LEADER
I20260926 11:29:58.201093 2859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:29:58.205412 2859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:29:58.211517 2859 catalog_manager.cc:1347] Generated new cluster ID: e5e0f8de0b2e4ad9befbf2817c0899e4
I20260926 11:29:58.211586 2859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:29:58.229027 2859 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:29:58.229969 2859 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:29:58.236068 2859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d8260d18ea34ac8a0c969296510df12: Generated new TSK 0
I20260926 11:29:58.236729 2859 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:29:58.272266 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41391
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:29:58.411031 2876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:29:58.411291 2876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:29:58.411334 2876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:29:58.415298 2876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:29:58.415350 2876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:29:58.415406 2876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:29:58.420713 2876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:29:58.421693 2876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:29:58.422708 2876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:29:58.431784 2881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:29:58.431864 2876 server_base.cc:1034] running on GCE node
W20260926 11:29:58.431761 2882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:29:58.431761 2884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:29:58.432462 2876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:29:58.433089 2876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:29:58.434266 2876 hybrid_clock.cc:648] HybridClock initialized: now 1790422198434255 us; error 28 us; skew 500 ppm
Sep 26 11:29:58 dist-test-slave-mrck krb5kdc[2769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:29:58.436897 2876 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:29:58.437676 2876 webserver.cc:460] Webserver started at http://127.0.98.1:35379/ using document root <none> and password file <none>
I20260926 11:29:58.438191 2876 fs_manager.cc:359] Metadata directory not provided
I20260926 11:29:58.438257 2876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:29:58.438407 2876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:29:58.440306 2876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/data/instance:
uuid: "d97c5dab8f274c358b386ef1322d2e13"
format_stamp: "Formatted at 2026-09-26 11:29:58 on dist-test-slave-mrck"
server_key: "f2ceaafc06a8af85215a987655c4a52e"
server_key_iv: "2e15a7bd254f6a2b5d2f79549a07b9af"
server_key_version: "encryptionkey@0"
I20260926 11:29:58.440786 2876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d97c5dab8f274c358b386ef1322d2e13"
format_stamp: "Formatted at 2026-09-26 11:29:58 on dist-test-slave-mrck"
server_key: "f2ceaafc06a8af85215a987655c4a52e"
server_key_iv: "2e15a7bd254f6a2b5d2f79549a07b9af"
server_key_version: "encryptionkey@0"
I20260926 11:29:58.444247 2876 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:29:58.446506 2891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:58.447703 2876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:29:58.447878 2876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "d97c5dab8f274c358b386ef1322d2e13"
format_stamp: "Formatted at 2026-09-26 11:29:58 on dist-test-slave-mrck"
server_key: "f2ceaafc06a8af85215a987655c4a52e"
server_key_iv: "2e15a7bd254f6a2b5d2f79549a07b9af"
server_key_version: "encryptionkey@0"
I20260926 11:29:58.447986 2876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:29:58.476910 2876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:29:58.479799 2876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:29:58.479996 2876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:29:58.480608 2876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:29:58.481407 2876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:29:58.481452 2876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:58.481490 2876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:29:58.481506 2876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:58.490834 2876 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35403
I20260926 11:29:58.490854 3004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35403 every 8 connection(s)
I20260926 11:29:58.491838 2876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:29:58.499171 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 2876
I20260926 11:29:58.499280 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:29:58.499619 392 external_mini_cluster.cc:1428] Setting key d8e480d62c8285af0b70b25c7fee8f04
Sep 26 11:29:58 dist-test-slave-mrck krb5kdc[2769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:29:58.505825 2853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:58.493550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53115 (local address 127.0.98.62:41391)
0926 11:29:58.493705 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:58.493709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:58.494513 (+ 804us) server_negotiation.cc:407] Connection header received
0926 11:29:58.495885 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:58.495887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:58.495933 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:58.496002 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:58.497103 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:58.497631 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:58.498686 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:58.498896 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:58.498971 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:58.501985 (+ 3014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:58.501988 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:58.501991 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:58.502022 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:58.503590 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:58.504264 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:58.504267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:58.504268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:58.504345 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:58.504729 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:58.504731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:58.504733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:58.504893 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:58.504967 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:58.505609 (+ 642us) server_negotiation.cc:299] Negotiation successful
0926 11:29:58.505709 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 11:29:58.507102 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:58.493937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41391 (local address 127.0.98.1:53115)
0926 11:29:58.494353 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:58.494413 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:58.495624 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:58.496151 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:58.496157 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:58.496505 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:29:58.496968 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:58.496980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:58.497731 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:58.497733 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:58.498582 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:58.498590 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:58.499049 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:58.499056 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:58.499270 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:58.500032 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:29:58.500054 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:29:58.501827 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:29:58.503755 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:58.503765 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:58.503782 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:58.504157 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:58.504440 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:58.504447 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:58.504452 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:58.504634 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:58.504997 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:29:58.505006 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:29:58.505343 (+ 337us) client_negotiation.cc:769] Sending connection context
0926 11:29:58.505561 (+ 218us) client_negotiation.cc:240] Negotiation successful
0926 11:29:58.505947 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":110,"threads_started":1}
I20260926 11:29:58.508322 3005 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41391
I20260926 11:29:58.508610 3005 heartbeater.cc:464] Registering TS with master...
I20260926 11:29:58.509223 3005 heartbeater.cc:511] Master 127.0.98.62:41391 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 11:29:58.510825 2812 ts_manager.cc:194] Registered new tserver with Master: d97c5dab8f274c358b386ef1322d2e13 (127.0.98.1:35403)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 11:29:58.512122 2812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53115
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:29:58.524662 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41391
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:29:58.630993 3010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:29:58.631224 3010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:29:58.631265 3010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:29:58.635198 3010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:29:58.635254 3010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:29:58.635342 3010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:29:58.640478 3010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:29:58.641422 3010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:29:58.642292 3010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:29:58.651485 3015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:29:58.651662 3018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:29:58.651743 3016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:29:58.651894 3010 server_base.cc:1034] running on GCE node
I20260926 11:29:58.652303 3010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:29:58.652889 3010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:29:58.654088 3010 hybrid_clock.cc:648] HybridClock initialized: now 1790422198654053 us; error 49 us; skew 500 ppm
Sep 26 11:29:58 dist-test-slave-mrck krb5kdc[2769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:29:58.656764 3010 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:29:58.657598 3010 webserver.cc:460] Webserver started at http://127.0.98.2:44269/ using document root <none> and password file <none>
I20260926 11:29:58.658160 3010 fs_manager.cc:359] Metadata directory not provided
I20260926 11:29:58.658262 3010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:29:58.658486 3010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:29:58.660542 3010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/data/instance:
uuid: "d1f883ba41384198b074efa0bdf00a8c"
format_stamp: "Formatted at 2026-09-26 11:29:58 on dist-test-slave-mrck"
server_key: "0bbb1d563ff632285ca4b8f851188bc9"
server_key_iv: "087cb2e593170fe3f75cab0650ea3b7d"
server_key_version: "encryptionkey@0"
I20260926 11:29:58.661108 3010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d1f883ba41384198b074efa0bdf00a8c"
format_stamp: "Formatted at 2026-09-26 11:29:58 on dist-test-slave-mrck"
server_key: "0bbb1d563ff632285ca4b8f851188bc9"
server_key_iv: "087cb2e593170fe3f75cab0650ea3b7d"
server_key_version: "encryptionkey@0"
I20260926 11:29:58.664683 3010 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:29:58.666973 3025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:58.668226 3010 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 11:29:58.668354 3010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "d1f883ba41384198b074efa0bdf00a8c"
format_stamp: "Formatted at 2026-09-26 11:29:58 on dist-test-slave-mrck"
server_key: "0bbb1d563ff632285ca4b8f851188bc9"
server_key_iv: "087cb2e593170fe3f75cab0650ea3b7d"
server_key_version: "encryptionkey@0"
I20260926 11:29:58.668458 3010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:29:58.681968 3010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:29:58.684743 3010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:29:58.684933 3010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:29:58.685484 3010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:29:58.686234 3010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:29:58.686301 3010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:58.686409 3010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:29:58.686458 3010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:58.695013 3010 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35341
I20260926 11:29:58.695034 3138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35341 every 8 connection(s)
I20260926 11:29:58.695925 3010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:29:58.700289 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 3010
I20260926 11:29:58.700431 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:29:58.700740 392 external_mini_cluster.cc:1428] Setting key 2191377c15dc1802768e92d27b32a1e3
Sep 26 11:29:58 dist-test-slave-mrck krb5kdc[2769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 11:29:58.711076 2853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:58.697685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44817 (local address 127.0.98.62:41391)
0926 11:29:58.697820 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:58.697824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:58.698634 (+ 810us) server_negotiation.cc:407] Connection header received
0926 11:29:58.700261 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:58.700265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:58.700359 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:58.700464 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:58.702106 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:58.702875 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:58.704096 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:58.704539 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:58.704699 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:58.707282 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:58.707286 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:58.707292 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:58.707325 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:58.709053 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:58.709554 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:58.709558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:58.709561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:58.709654 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:58.709977 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:58.709983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:58.709987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:58.710239 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:58.710337 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:58.710859 (+ 522us) server_negotiation.cc:299] Negotiation successful
0926 11:29:58.710956 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 11:29:58.711978 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:58.698033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41391 (local address 127.0.98.2:44817)
0926 11:29:58.698489 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:58.698530 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:58.699964 (+ 1434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:58.700836 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:58.700846 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:58.701298 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:29:58.701962 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:58.701988 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:58.703013 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:58.703016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:58.703954 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:58.703964 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:58.704740 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:58.704743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:58.704897 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:58.705532 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:29:58.705547 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:29:58.707115 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:29:58.709164 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:58.709168 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:58.709180 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:58.709449 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:58.709752 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:58.709755 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:58.709756 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:58.709879 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:58.710334 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:29:58.710338 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:29:58.710620 (+ 282us) client_negotiation.cc:769] Sending connection context
0926 11:29:58.710775 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 11:29:58.711015 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 11:29:58.713279 3139 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41391
I20260926 11:29:58.713558 3139 heartbeater.cc:464] Registering TS with master...
I20260926 11:29:58.714210 3139 heartbeater.cc:511] Master 127.0.98.62:41391 requested a full tablet report, sending...
I20260926 11:29:58.715312 2812 ts_manager.cc:194] Registered new tserver with Master: d1f883ba41384198b074efa0bdf00a8c (127.0.98.2:35341)
I20260926 11:29:58.716022 2812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44817
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:29:58.730676 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41391
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:29:58.838867 3144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:29:58.839141 3144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:29:58.839210 3144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:29:58.843169 3144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:29:58.843254 3144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:29:58.843343 3144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:29:58.848690 3144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41391
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:29:58.849709 3144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:29:58.850715 3144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:29:58.860518 3150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:29:58.860560 3149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:29:58.860697 3144 server_base.cc:1034] running on GCE node
W20260926 11:29:58.860879 3152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:29:58.861266 3144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:29:58.861855 3144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:29:58.863055 3144 hybrid_clock.cc:648] HybridClock initialized: now 1790422198863036 us; error 27 us; skew 500 ppm
Sep 26 11:29:58 dist-test-slave-mrck krb5kdc[2769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:29:58.865746 3144 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:29:58.866631 3144 webserver.cc:460] Webserver started at http://127.0.98.3:39457/ using document root <none> and password file <none>
I20260926 11:29:58.867200 3144 fs_manager.cc:359] Metadata directory not provided
I20260926 11:29:58.867290 3144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:29:58.867517 3144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:29:58.869706 3144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/data/instance:
uuid: "275f93dd27354c288f0c729155fda414"
format_stamp: "Formatted at 2026-09-26 11:29:58 on dist-test-slave-mrck"
server_key: "81ac00eb6fd407ba36dba74fd9bdee04"
server_key_iv: "0712b9bd4b1fda8a1a9ad7ea2d6373dd"
server_key_version: "encryptionkey@0"
I20260926 11:29:58.870291 3144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "275f93dd27354c288f0c729155fda414"
format_stamp: "Formatted at 2026-09-26 11:29:58 on dist-test-slave-mrck"
server_key: "81ac00eb6fd407ba36dba74fd9bdee04"
server_key_iv: "0712b9bd4b1fda8a1a9ad7ea2d6373dd"
server_key_version: "encryptionkey@0"
I20260926 11:29:58.874126 3144 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 11:29:58.876679 3159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:58.877949 3144 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:29:58.878087 3144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "275f93dd27354c288f0c729155fda414"
format_stamp: "Formatted at 2026-09-26 11:29:58 on dist-test-slave-mrck"
server_key: "81ac00eb6fd407ba36dba74fd9bdee04"
server_key_iv: "0712b9bd4b1fda8a1a9ad7ea2d6373dd"
server_key_version: "encryptionkey@0"
I20260926 11:29:58.878201 3144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:29:58.899432 3144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:29:58.902896 3144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:29:58.903110 3144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:29:58.903710 3144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:29:58.904614 3144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:29:58.904686 3144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:58.904758 3144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:29:58.904799 3144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:29:58.913650 3144 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39807
I20260926 11:29:58.913676 3272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39807 every 8 connection(s)
I20260926 11:29:58.914547 3144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:29:58.916821 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 3144
I20260926 11:29:58.916942 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:29:58.917317 392 external_mini_cluster.cc:1428] Setting key ab862ac145fe2d901cf18d65f397c42e
Sep 26 11:29:58 dist-test-slave-mrck krb5kdc[2769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:29:58.927687 2853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:58.916457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46967 (local address 127.0.98.62:41391)
0926 11:29:58.916629 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:58.916635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:58.917573 (+ 938us) server_negotiation.cc:407] Connection header received
0926 11:29:58.919120 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:58.919123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:58.919202 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:58.919291 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:58.920544 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:58.921080 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:58.922180 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:29:58.922377 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:29:58.922454 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:58.924826 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:29:58.924828 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:29:58.924830 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:29:58.924851 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:29:58.926029 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:58.926445 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:58.926447 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:58.926448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:58.926517 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:29:58.926788 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:29:58.926790 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:29:58.926791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:29:58.926957 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:29:58.927029 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 11:29:58.927478 (+ 449us) server_negotiation.cc:299] Negotiation successful
0926 11:29:58.927570 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 11:29:58.928524 3275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:29:58.916822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41391 (local address 127.0.98.3:46967)
0926 11:29:58.917397 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0926 11:29:58.917455 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 11:29:58.918921 (+ 1466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:29:58.919495 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:29:58.919502 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:29:58.919931 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:29:58.920421 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:58.920435 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:58.921190 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:58.921193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:58.922068 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:29:58.922076 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:29:58.922492 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:29:58.922495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:29:58.922627 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:29:58.923224 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:29:58.923240 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:29:58.924687 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:29:58.926122 (+ 1435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:58.926126 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:58.926137 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:58.926364 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:58.926596 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:29:58.926598 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:29:58.926599 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:29:58.926717 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:29:58.927052 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:29:58.927057 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:29:58.927286 (+ 229us) client_negotiation.cc:769] Sending connection context
0926 11:29:58.927437 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 11:29:58.927668 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":148,"threads_started":1}
I20260926 11:29:58.929646 3273 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41391
I20260926 11:29:58.929929 3273 heartbeater.cc:464] Registering TS with master...
I20260926 11:29:58.930487 3273 heartbeater.cc:511] Master 127.0.98.62:41391 requested a full tablet report, sending...
I20260926 11:29:58.931519 2812 ts_manager.cc:194] Registered new tserver with Master: 275f93dd27354c288f0c729155fda414 (127.0.98.3:39807)
I20260926 11:29:58.932089 2812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46967
I20260926 11:29:58.942710 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 11:29:58.954926 2853 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:29:58.953631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60674 (local address 127.0.98.62:41391)
0926 11:29:58.953750 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 11:29:58.953753 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:29:58.953916 (+ 163us) server_negotiation.cc:407] Connection header received
0926 11:29:58.954020 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:29:58.954022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:29:58.954061 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:29:58.954135 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:29:58.954779 (+ 644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60674: BlockingRecv error: recv got EOF from 127.0.0.1:60674 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 11:29:58.956020 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 2876
I20260926 11:29:58.961254 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 3010
I20260926 11:29:58.966140 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 3144
I20260926 11:29:58.971122 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 2782
2026-09-26T11:29:58Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2101 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:29:59 dist-test-slave-mrck krb5kdc[3285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:29:59 dist-test-slave-mrck krb5kdc[3285](info): set up 2 sockets
Sep 26 11:29:59 dist-test-slave-mrck krb5kdc[3285](info): commencing operation
krb5kdc: starting...
W20260926 11:30:00.007679 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:30:00 dist-test-slave-mrck krb5kdc[3285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:30:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:30:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:30:00.128216 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44259
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:00.235510 3298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:00.235875 3298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:00.235966 3298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:00.240612 3298 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:30:00.240703 3298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:00.240736 3298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:00.240777 3298 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:30:00.240809 3298 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:30:00.246199 3298 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:00.247313 3298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:00.248394 3298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:00.256373 3306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:00.256505 3298 server_base.cc:1034] running on GCE node
W20260926 11:30:00.256386 3303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:00.256407 3304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:00.257185 3298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:00.258296 3298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:00.259487 3298 hybrid_clock.cc:648] HybridClock initialized: now 1790422200259455 us; error 45 us; skew 500 ppm
Sep 26 11:30:00 dist-test-slave-mrck krb5kdc[3285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:00.268210 3298 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:30:00.269184 3298 webserver.cc:460] Webserver started at http://127.0.98.62:33235/ using document root <none> and password file <none>
I20260926 11:30:00.269749 3298 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:00.269838 3298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:00.270056 3298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:00.272147 3298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "b1673423f350418aa0e8ed1afc849da8"
format_stamp: "Formatted at 2026-09-26 11:30:00 on dist-test-slave-mrck"
server_key: "219ad60cf5a93a5fbcdfa363673512b0"
server_key_iv: "ab0649ea20c5fcec0ced56bc0e7d0b95"
server_key_version: "encryptionkey@0"
I20260926 11:30:00.272708 3298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "b1673423f350418aa0e8ed1afc849da8"
format_stamp: "Formatted at 2026-09-26 11:30:00 on dist-test-slave-mrck"
server_key: "219ad60cf5a93a5fbcdfa363673512b0"
server_key_iv: "ab0649ea20c5fcec0ced56bc0e7d0b95"
server_key_version: "encryptionkey@0"
I20260926 11:30:00.276425 3298 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:30:00.278752 3313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:00.280138 3298 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260926 11:30:00.280282 3298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "b1673423f350418aa0e8ed1afc849da8"
format_stamp: "Formatted at 2026-09-26 11:30:00 on dist-test-slave-mrck"
server_key: "219ad60cf5a93a5fbcdfa363673512b0"
server_key_iv: "ab0649ea20c5fcec0ced56bc0e7d0b95"
server_key_version: "encryptionkey@0"
I20260926 11:30:00.280406 3298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:00.294837 3298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:00.298171 3298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:00.298390 3298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:00.305538 3298 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44259
I20260926 11:30:00.305845 3365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44259 every 8 connection(s)
I20260926 11:30:00.306545 3298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:30:00.309355 3366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:00.314055 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 3298
I20260926 11:30:00.314213 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:30:00.314484 392 external_mini_cluster.cc:1428] Setting key 0bb0fc26df83107596f589494d1f389a
I20260926 11:30:00.315059 3366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1673423f350418aa0e8ed1afc849da8: Bootstrap starting.
I20260926 11:30:00.317744 3366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1673423f350418aa0e8ed1afc849da8: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:00.318609 3366 log.cc:824] T 00000000000000000000000000000000 P b1673423f350418aa0e8ed1afc849da8: Log is configured to *not* fsync() on all Append() calls
I20260926 11:30:00.320696 3366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1673423f350418aa0e8ed1afc849da8: No bootstrap required, opened a new log
Sep 26 11:30:00 dist-test-slave-mrck krb5kdc[3285](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:30:00.324559 3366 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1673423f350418aa0e8ed1afc849da8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1673423f350418aa0e8ed1afc849da8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44259 } }
I20260926 11:30:00.324856 3366 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1673423f350418aa0e8ed1afc849da8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:00.324944 3366 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1673423f350418aa0e8ed1afc849da8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1673423f350418aa0e8ed1afc849da8, State: Initialized, Role: FOLLOWER
I20260926 11:30:00.325496 3366 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1673423f350418aa0e8ed1afc849da8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1673423f350418aa0e8ed1afc849da8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44259 } }
I20260926 11:30:00.325670 3366 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1673423f350418aa0e8ed1afc849da8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:30:00.325737 3366 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1673423f350418aa0e8ed1afc849da8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:30:00.325845 3366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1673423f350418aa0e8ed1afc849da8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:00.327005 3366 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1673423f350418aa0e8ed1afc849da8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1673423f350418aa0e8ed1afc849da8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44259 } }
I20260926 11:30:00.327409 3366 leader_election.cc:302] T 00000000000000000000000000000000 P b1673423f350418aa0e8ed1afc849da8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1673423f350418aa0e8ed1afc849da8; no voters:
I20260926 11:30:00.327752 3366 leader_election.cc:288] T 00000000000000000000000000000000 P b1673423f350418aa0e8ed1afc849da8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:30:00.327901 3371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1673423f350418aa0e8ed1afc849da8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:30:00.328143 3371 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1673423f350418aa0e8ed1afc849da8 [term 1 LEADER]: Becoming Leader. State: Replica: b1673423f350418aa0e8ed1afc849da8, State: Running, Role: LEADER
I20260926 11:30:00.328464 3371 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1673423f350418aa0e8ed1afc849da8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1673423f350418aa0e8ed1afc849da8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44259 } }
I20260926 11:30:00.329022 3366 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1673423f350418aa0e8ed1afc849da8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:30:00.329659 3371 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1673423f350418aa0e8ed1afc849da8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1673423f350418aa0e8ed1afc849da8. Latest consensus state: current_term: 1 leader_uuid: "b1673423f350418aa0e8ed1afc849da8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1673423f350418aa0e8ed1afc849da8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44259 } } }
I20260926 11:30:00.329757 3371 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1673423f350418aa0e8ed1afc849da8 [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:00.330020 3369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:00.315940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35172 (local address 127.0.98.62:44259)
0926 11:30:00.316386 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:00.316405 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:00.316446 (+ 41us) server_negotiation.cc:407] Connection header received
0926 11:30:00.317253 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:00.317276 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:00.317688 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:00.318084 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:00.318735 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:00.320092 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:00.321067 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:00.321721 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:00.321890 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:00.324364 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:00.324370 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:00.324387 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:00.324415 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:00.326466 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:00.326876 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:00.326885 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:00.326894 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:00.327037 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:00.327331 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:00.327334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:00.327336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:00.327898 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:00.328109 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:00.328738 (+ 629us) server_negotiation.cc:299] Negotiation successful
0926 11:30:00.329027 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":103,"threads_started":1}
I20260926 11:30:00.330116 3372 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1673423f350418aa0e8ed1afc849da8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1673423f350418aa0e8ed1afc849da8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1673423f350418aa0e8ed1afc849da8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44259 } } }
I20260926 11:30:00.330207 3372 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1673423f350418aa0e8ed1afc849da8 [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:00.333227 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260926 11:30:00.333904 3386 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b1673423f350418aa0e8ed1afc849da8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:30:00.333968 3386 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:30:00.334075 3379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:30:00.334920 3379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:30:00.344060 3379 catalog_manager.cc:1347] Generated new cluster ID: 1fca272a0c17486f9acde5a1bd41b42f
I20260926 11:30:00.344125 3379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:30:00.353926 3369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:00.348171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35188 (local address 127.0.98.62:44259)
0926 11:30:00.348280 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:00.348284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:00.348568 (+ 284us) server_negotiation.cc:407] Connection header received
0926 11:30:00.348694 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:00.348697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:00.348744 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:00.348823 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:00.349419 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:00.350233 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:00.351510 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:00.351888 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:00.351978 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:00.352906 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:00.352910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:00.352913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:30:00.353469 (+ 556us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:00.353499 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:30:00.353517 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:00.353597 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:00.353712 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 11:30:00.353771 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 11:30:00.360250 3379 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:30:00.361169 3379 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:30:00.366513 3379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1673423f350418aa0e8ed1afc849da8: Generated new TSK 0
I20260926 11:30:00.367164 3379 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:30:00.373801 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 3298
2026-09-26T11:30:00Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1400 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:30:00 dist-test-slave-mrck krb5kdc[3399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:30:00 dist-test-slave-mrck krb5kdc[3399](info): set up 2 sockets
Sep 26 11:30:00 dist-test-slave-mrck krb5kdc[3399](info): commencing operation
krb5kdc: starting...
W20260926 11:30:01.403657 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:30:01 dist-test-slave-mrck krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:30:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:30:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:30:01.518392 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33517
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:01.625483 3412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:01.625769 3412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:01.625810 3412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:01.629966 3412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:30:01.630029 3412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:01.630048 3412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:01.630065 3412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:30:01.630131 3412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:30:01.635438 3412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:01.636523 3412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:01.637549 3412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:01.645870 3418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:01.645859 3417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:01.645874 3420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:01.646250 3412 server_base.cc:1034] running on GCE node
I20260926 11:30:01.646857 3412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:01.648061 3412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:01.649215 3412 hybrid_clock.cc:648] HybridClock initialized: now 1790422201649189 us; error 56 us; skew 500 ppm
Sep 26 11:30:01 dist-test-slave-mrck krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:01.651923 3412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:30:01.652796 3412 webserver.cc:460] Webserver started at http://127.0.98.62:39399/ using document root <none> and password file <none>
I20260926 11:30:01.653363 3412 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:01.653450 3412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:01.653674 3412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:01.655637 3412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "d7613c766a6b46c1b7bf0eaaa0ed9e84"
format_stamp: "Formatted at 2026-09-26 11:30:01 on dist-test-slave-mrck"
server_key: "ebc2b3b8ec2063e08beb8b1e5a842dc2"
server_key_iv: "31591204d76f4a1652cce205e11fd84d"
server_key_version: "encryptionkey@0"
I20260926 11:30:01.656265 3412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "d7613c766a6b46c1b7bf0eaaa0ed9e84"
format_stamp: "Formatted at 2026-09-26 11:30:01 on dist-test-slave-mrck"
server_key: "ebc2b3b8ec2063e08beb8b1e5a842dc2"
server_key_iv: "31591204d76f4a1652cce205e11fd84d"
server_key_version: "encryptionkey@0"
I20260926 11:30:01.659855 3412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:30:01.662159 3427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:01.663354 3412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:30:01.663499 3412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "d7613c766a6b46c1b7bf0eaaa0ed9e84"
format_stamp: "Formatted at 2026-09-26 11:30:01 on dist-test-slave-mrck"
server_key: "ebc2b3b8ec2063e08beb8b1e5a842dc2"
server_key_iv: "31591204d76f4a1652cce205e11fd84d"
server_key_version: "encryptionkey@0"
I20260926 11:30:01.663614 3412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:01.686064 3412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:01.689210 3412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:01.689435 3412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:01.696362 3412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33517
I20260926 11:30:01.696642 3479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33517 every 8 connection(s)
I20260926 11:30:01.697382 3412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:30:01.700209 3480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:01.704288 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 3412
I20260926 11:30:01.704440 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:30:01.704704 392 external_mini_cluster.cc:1428] Setting key c1e89992c60a49caa1c1a13470ae07e8
I20260926 11:30:01.705834 3480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7613c766a6b46c1b7bf0eaaa0ed9e84: Bootstrap starting.
I20260926 11:30:01.708137 3480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7613c766a6b46c1b7bf0eaaa0ed9e84: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:01.708973 3480 log.cc:824] T 00000000000000000000000000000000 P d7613c766a6b46c1b7bf0eaaa0ed9e84: Log is configured to *not* fsync() on all Append() calls
I20260926 11:30:01.711004 3480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7613c766a6b46c1b7bf0eaaa0ed9e84: No bootstrap required, opened a new log
Sep 26 11:30:01 dist-test-slave-mrck krb5kdc[3399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:30:01.715109 3480 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7613c766a6b46c1b7bf0eaaa0ed9e84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7613c766a6b46c1b7bf0eaaa0ed9e84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33517 } }
I20260926 11:30:01.715315 3480 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7613c766a6b46c1b7bf0eaaa0ed9e84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:01.715399 3480 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7613c766a6b46c1b7bf0eaaa0ed9e84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7613c766a6b46c1b7bf0eaaa0ed9e84, State: Initialized, Role: FOLLOWER
I20260926 11:30:01.715926 3480 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7613c766a6b46c1b7bf0eaaa0ed9e84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7613c766a6b46c1b7bf0eaaa0ed9e84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33517 } }
I20260926 11:30:01.716092 3480 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7613c766a6b46c1b7bf0eaaa0ed9e84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:30:01.716166 3480 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7613c766a6b46c1b7bf0eaaa0ed9e84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:30:01.716265 3480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7613c766a6b46c1b7bf0eaaa0ed9e84 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:01.717356 3480 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7613c766a6b46c1b7bf0eaaa0ed9e84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7613c766a6b46c1b7bf0eaaa0ed9e84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33517 } }
I20260926 11:30:01.717707 3480 leader_election.cc:302] T 00000000000000000000000000000000 P d7613c766a6b46c1b7bf0eaaa0ed9e84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7613c766a6b46c1b7bf0eaaa0ed9e84; no voters:
I20260926 11:30:01.718053 3480 leader_election.cc:288] T 00000000000000000000000000000000 P d7613c766a6b46c1b7bf0eaaa0ed9e84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:30:01.718144 3485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7613c766a6b46c1b7bf0eaaa0ed9e84 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:30:01.718437 3485 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7613c766a6b46c1b7bf0eaaa0ed9e84 [term 1 LEADER]: Becoming Leader. State: Replica: d7613c766a6b46c1b7bf0eaaa0ed9e84, State: Running, Role: LEADER
I20260926 11:30:01.718794 3485 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7613c766a6b46c1b7bf0eaaa0ed9e84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7613c766a6b46c1b7bf0eaaa0ed9e84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33517 } }
I20260926 11:30:01.719343 3480 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7613c766a6b46c1b7bf0eaaa0ed9e84 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:30:01.719425 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:01.706000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59430 (local address 127.0.98.62:33517)
0926 11:30:01.706400 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:01.706422 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:01.706456 (+ 34us) server_negotiation.cc:407] Connection header received
0926 11:30:01.707071 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:01.707088 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:01.707413 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:01.707685 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:01.708633 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:01.709840 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:01.710933 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:01.711429 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:01.711543 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:01.714260 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:01.714265 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:01.714278 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:01.714304 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:01.716377 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:01.716785 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:01.716790 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:01.716797 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:01.716930 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:01.717190 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:01.717196 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:01.717201 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:01.717712 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:01.717889 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:01.718202 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 11:30:01.718495 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":130,"threads_started":1}
I20260926 11:30:01.721276 3486 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7613c766a6b46c1b7bf0eaaa0ed9e84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7613c766a6b46c1b7bf0eaaa0ed9e84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7613c766a6b46c1b7bf0eaaa0ed9e84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33517 } } }
I20260926 11:30:01.721408 3486 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7613c766a6b46c1b7bf0eaaa0ed9e84 [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:01.721573 3487 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7613c766a6b46c1b7bf0eaaa0ed9e84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7613c766a6b46c1b7bf0eaaa0ed9e84. Latest consensus state: current_term: 1 leader_uuid: "d7613c766a6b46c1b7bf0eaaa0ed9e84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7613c766a6b46c1b7bf0eaaa0ed9e84" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33517 } } }
I20260926 11:30:01.722039 3487 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7613c766a6b46c1b7bf0eaaa0ed9e84 [sys.catalog]: This master's current role is: LEADER
W20260926 11:30:01.724095 3499 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d7613c766a6b46c1b7bf0eaaa0ed9e84: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:30:01.724153 3499 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:30:01.724259 3500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:30:01.725320 3500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:30:01.730895 3500 catalog_manager.cc:1347] Generated new cluster ID: 58cad72f621941aea6974c48323aa60e
I20260926 11:30:01.730957 3500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:30:01.742636 3500 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:30:01.743950 3500 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:30:01.753029 3500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7613c766a6b46c1b7bf0eaaa0ed9e84: Generated new TSK 0
I20260926 11:30:01.753938 3500 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:30:01.792697 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33517
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:01.906002 3506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:01.906261 3506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:01.906301 3506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:01.910308 3506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:01.910367 3506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:01.910423 3506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:30:01.915499 3506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:01.916545 3506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:01.917465 3506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:01.926893 3511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:01.926950 3506 server_base.cc:1034] running on GCE node
W20260926 11:30:01.926981 3514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:01.926903 3512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:01.927464 3506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:01.928125 3506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:01.929301 3506 hybrid_clock.cc:648] HybridClock initialized: now 1790422201929270 us; error 43 us; skew 500 ppm
Sep 26 11:30:01 dist-test-slave-mrck krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:01.931898 3506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:30:01.932690 3506 webserver.cc:460] Webserver started at http://127.0.98.1:33083/ using document root <none> and password file <none>
I20260926 11:30:01.933241 3506 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:01.933328 3506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:01.933553 3506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:01.935459 3506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/data/instance:
uuid: "f571fa3b10ac4023b8242e7b86485eb5"
format_stamp: "Formatted at 2026-09-26 11:30:01 on dist-test-slave-mrck"
server_key: "0e89e50fdbe5c161974028cf15f93256"
server_key_iv: "011614008c2922cde381e39e9fd793dc"
server_key_version: "encryptionkey@0"
I20260926 11:30:01.936008 3506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f571fa3b10ac4023b8242e7b86485eb5"
format_stamp: "Formatted at 2026-09-26 11:30:01 on dist-test-slave-mrck"
server_key: "0e89e50fdbe5c161974028cf15f93256"
server_key_iv: "011614008c2922cde381e39e9fd793dc"
server_key_version: "encryptionkey@0"
I20260926 11:30:01.939345 3506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 11:30:01.941677 3521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:01.942811 3506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 11:30:01.942938 3506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "f571fa3b10ac4023b8242e7b86485eb5"
format_stamp: "Formatted at 2026-09-26 11:30:01 on dist-test-slave-mrck"
server_key: "0e89e50fdbe5c161974028cf15f93256"
server_key_iv: "011614008c2922cde381e39e9fd793dc"
server_key_version: "encryptionkey@0"
I20260926 11:30:01.943050 3506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:01.955888 3506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:01.958530 3506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:01.958719 3506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:01.959272 3506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:01.960156 3506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:01.960225 3506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:01.960294 3506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:01.960346 3506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:01.968968 3506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34633
I20260926 11:30:01.968987 3634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34633 every 8 connection(s)
I20260926 11:30:01.969801 3506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 11:30:01 dist-test-slave-mrck krb5kdc[3399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:30:01.979439 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 3506
I20260926 11:30:01.979576 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:30:01.979911 392 external_mini_cluster.cc:1428] Setting key 24a3cf25f1cfeb4bbd6a02e53fd3187c
I20260926 11:30:01.982707 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:01.971483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50475 (local address 127.0.98.62:33517)
0926 11:30:01.971652 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:01.971658 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:01.972541 (+ 883us) server_negotiation.cc:407] Connection header received
0926 11:30:01.973802 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:01.973804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:01.973846 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:01.973919 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:01.974994 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:01.975513 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:01.976756 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:01.976971 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:01.977043 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:01.979247 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:01.979250 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:01.979252 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:01.979274 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:01.980776 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:01.981260 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:01.981266 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:01.981268 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:01.981345 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:01.981678 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:01.981683 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:01.981685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:01.981855 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:01.981963 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:01.982429 (+ 466us) server_negotiation.cc:299] Negotiation successful
0926 11:30:01.982543 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260926 11:30:01.983520 3637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:01.971980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33517 (local address 127.0.98.1:50475)
0926 11:30:01.972400 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:01.972440 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:01.973625 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:01.974074 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:01.974080 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:01.974416 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:01.974877 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:01.974891 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:01.975626 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:01.975630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:01.976648 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:01.976657 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:01.977086 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:01.977088 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:01.977217 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:01.977728 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:01.977743 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:01.979104 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:01.980886 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:01.980892 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:01.980915 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:01.981162 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:01.981453 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:01.981456 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:01.981458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:01.981588 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:01.981960 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:01.981967 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:01.982200 (+ 233us) client_negotiation.cc:769] Sending connection context
0926 11:30:01.982384 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 11:30:01.982687 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":108,"threads_started":1}
I20260926 11:30:01.984723 3635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33517
I20260926 11:30:01.984987 3635 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:01.985569 3635 heartbeater.cc:511] Master 127.0.98.62:33517 requested a full tablet report, sending...
I20260926 11:30:01.987274 3444 ts_manager.cc:194] Registered new tserver with Master: f571fa3b10ac4023b8242e7b86485eb5 (127.0.98.1:34633)
I20260926 11:30:01.988945 3444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50475
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:30:02.007579 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33517
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:02.117254 3640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:02.117524 3640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:02.117568 3640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:02.121714 3640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:02.121766 3640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:02.121830 3640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:30:02.127038 3640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:02.127971 3640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:02.128892 3640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:02.137960 3648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:02.137948 3645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:02.137964 3646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:02.138159 3640 server_base.cc:1034] running on GCE node
I20260926 11:30:02.138757 3640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:02.139376 3640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:02.140555 3640 hybrid_clock.cc:648] HybridClock initialized: now 1790422202140542 us; error 31 us; skew 500 ppm
Sep 26 11:30:02 dist-test-slave-mrck krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:02.143147 3640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:30:02.144001 3640 webserver.cc:460] Webserver started at http://127.0.98.2:46419/ using document root <none> and password file <none>
I20260926 11:30:02.144500 3640 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:02.144554 3640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:02.144716 3640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:02.146703 3640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/data/instance:
uuid: "f8d45350f8af4a6a91800c0bf5340ab3"
format_stamp: "Formatted at 2026-09-26 11:30:02 on dist-test-slave-mrck"
server_key: "4de96e8e0945c72d31a393f6cdf4eab3"
server_key_iv: "ef9e62228669f5ecb0e858acba8afc2c"
server_key_version: "encryptionkey@0"
I20260926 11:30:02.147219 3640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f8d45350f8af4a6a91800c0bf5340ab3"
format_stamp: "Formatted at 2026-09-26 11:30:02 on dist-test-slave-mrck"
server_key: "4de96e8e0945c72d31a393f6cdf4eab3"
server_key_iv: "ef9e62228669f5ecb0e858acba8afc2c"
server_key_version: "encryptionkey@0"
I20260926 11:30:02.150722 3640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 11:30:02.153012 3655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:02.154130 3640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 11:30:02.154247 3640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "f8d45350f8af4a6a91800c0bf5340ab3"
format_stamp: "Formatted at 2026-09-26 11:30:02 on dist-test-slave-mrck"
server_key: "4de96e8e0945c72d31a393f6cdf4eab3"
server_key_iv: "ef9e62228669f5ecb0e858acba8afc2c"
server_key_version: "encryptionkey@0"
I20260926 11:30:02.154342 3640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:02.167579 3640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:02.170532 3640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:02.170732 3640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:02.171439 3640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:02.172247 3640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:02.172293 3640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:02.172330 3640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:02.172351 3640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:02.180881 3640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45395
I20260926 11:30:02.180898 3768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45395 every 8 connection(s)
I20260926 11:30:02.181792 3640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:30:02.183471 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 3640
I20260926 11:30:02.183624 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:30:02.184039 392 external_mini_cluster.cc:1428] Setting key 67c344a4236fed071b89b9dce7dec099
Sep 26 11:30:02 dist-test-slave-mrck krb5kdc[3399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 11:30:02.197263 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:02.183564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49095 (local address 127.0.98.62:33517)
0926 11:30:02.183705 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:02.183709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:02.184964 (+ 1255us) server_negotiation.cc:407] Connection header received
0926 11:30:02.186910 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:02.186912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:02.186957 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:02.187029 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:02.188359 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:02.188883 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:02.190089 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:02.190316 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:02.190433 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:02.193235 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:02.193241 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:02.193247 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:02.193278 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:02.194787 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:02.195405 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:02.195411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:02.195416 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:02.195508 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:02.195992 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:02.195996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:02.195999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:02.196279 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:02.196403 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:02.196878 (+ 475us) server_negotiation.cc:299] Negotiation successful
0926 11:30:02.197030 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 11:30:02.198151 3771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:02.184111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33517 (local address 127.0.98.2:49095)
0926 11:30:02.184823 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:02.184872 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:02.186674 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:02.187213 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:02.187226 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:02.187647 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:02.188211 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:02.188224 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:02.189014 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:02.189018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:02.189967 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:02.189975 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:02.190466 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:02.190470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:02.190640 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:02.191218 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:02.191232 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:02.193061 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:02.194981 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:02.194986 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:02.194998 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:02.195249 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:02.195660 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:02.195667 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:02.195672 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:02.195879 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:02.196388 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:02.196393 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:02.196615 (+ 222us) client_negotiation.cc:769] Sending connection context
0926 11:30:02.196766 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 11:30:02.197061 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":110,"threads_started":1}
I20260926 11:30:02.199347 3769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33517
I20260926 11:30:02.199605 3769 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:02.200139 3769 heartbeater.cc:511] Master 127.0.98.62:33517 requested a full tablet report, sending...
I20260926 11:30:02.201332 3444 ts_manager.cc:194] Registered new tserver with Master: f8d45350f8af4a6a91800c0bf5340ab3 (127.0.98.2:45395)
I20260926 11:30:02.201918 3444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49095
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:30:02.211079 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33517
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:02.319214 3774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:02.319556 3774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:02.319645 3774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:02.324054 3774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:02.324136 3774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:02.324224 3774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:30:02.329636 3774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:02.330756 3774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:02.331874 3774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:02.341441 3779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:02.341511 3780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:02.341470 3782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:02.341929 3774 server_base.cc:1034] running on GCE node
I20260926 11:30:02.342320 3774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:02.342983 3774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:02.344157 3774 hybrid_clock.cc:648] HybridClock initialized: now 1790422202344126 us; error 52 us; skew 500 ppm
Sep 26 11:30:02 dist-test-slave-mrck krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:02.346909 3774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:30:02.347894 3774 webserver.cc:460] Webserver started at http://127.0.98.3:39357/ using document root <none> and password file <none>
I20260926 11:30:02.348454 3774 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:02.348542 3774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:02.348771 3774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:02.350739 3774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/data/instance:
uuid: "4590b2fe9c38429596eaeb743eef728e"
format_stamp: "Formatted at 2026-09-26 11:30:02 on dist-test-slave-mrck"
server_key: "3b1765e16390fea63bfa6621c18f452a"
server_key_iv: "813c6e9b33a7cccec2b6578c52763426"
server_key_version: "encryptionkey@0"
I20260926 11:30:02.351323 3774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4590b2fe9c38429596eaeb743eef728e"
format_stamp: "Formatted at 2026-09-26 11:30:02 on dist-test-slave-mrck"
server_key: "3b1765e16390fea63bfa6621c18f452a"
server_key_iv: "813c6e9b33a7cccec2b6578c52763426"
server_key_version: "encryptionkey@0"
I20260926 11:30:02.354883 3774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260926 11:30:02.357302 3789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:02.358538 3774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 11:30:02.358682 3774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "4590b2fe9c38429596eaeb743eef728e"
format_stamp: "Formatted at 2026-09-26 11:30:02 on dist-test-slave-mrck"
server_key: "3b1765e16390fea63bfa6621c18f452a"
server_key_iv: "813c6e9b33a7cccec2b6578c52763426"
server_key_version: "encryptionkey@0"
I20260926 11:30:02.358788 3774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:02.376264 3774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:02.379163 3774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:02.379367 3774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:02.380038 3774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:02.380838 3774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:02.380908 3774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:02.380980 3774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:02.381022 3774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:02.390069 3774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37925
I20260926 11:30:02.390090 3902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37925 every 8 connection(s)
I20260926 11:30:02.390940 3774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:30:02.397413 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 3774
I20260926 11:30:02.397531 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790422180210734-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:30:02.397804 392 external_mini_cluster.cc:1428] Setting key 113d4fcb49bad48c11d04c0beba56f00
Sep 26 11:30:02 dist-test-slave-mrck krb5kdc[3399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422202, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:30:02.404193 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:02.392806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56987 (local address 127.0.98.62:33517)
0926 11:30:02.392943 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:02.392946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:02.393747 (+ 801us) server_negotiation.cc:407] Connection header received
0926 11:30:02.395097 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:02.395100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:02.395152 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:02.395239 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:02.396383 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:02.396917 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:02.398456 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:02.398786 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:02.398922 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:02.401326 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:02.401329 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:02.401331 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:02.401353 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:02.402525 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:02.402949 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:02.402951 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:02.402952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:02.403017 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:02.403300 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:02.403302 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:02.403303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:02.403451 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:02.403521 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:02.403992 (+ 471us) server_negotiation.cc:299] Negotiation successful
0926 11:30:02.404082 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 11:30:02.404929 3905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:02.393184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33517 (local address 127.0.98.3:56987)
0926 11:30:02.393609 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:02.393647 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:02.394867 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:02.395421 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:02.395427 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:02.395790 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:02.396238 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:02.396251 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:02.397036 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:02.397039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:02.398324 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:02.398334 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:02.398967 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:02.398971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:02.399179 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:02.399824 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:02.399859 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:02.401189 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:02.402616 (+ 1427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:02.402620 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:02.402631 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:02.402872 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:02.403097 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:02.403099 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:02.403100 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:02.403228 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:02.403528 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:02.403532 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:02.403778 (+ 246us) client_negotiation.cc:769] Sending connection context
0926 11:30:02.403950 (+ 172us) client_negotiation.cc:240] Negotiation successful
0926 11:30:02.404176 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":129,"threads_started":1}
I20260926 11:30:02.405897 3903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33517
I20260926 11:30:02.406111 3903 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:02.406550 3903 heartbeater.cc:511] Master 127.0.98.62:33517 requested a full tablet report, sending...
I20260926 11:30:02.407482 3444 ts_manager.cc:194] Registered new tserver with Master: 4590b2fe9c38429596eaeb743eef728e (127.0.98.3:37925)
I20260926 11:30:02.408068 3444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56987
I20260926 11:30:02.411186 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 11:30:02 dist-test-slave-mrck krb5kdc[3399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422202, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 11:30:02 dist-test-slave-mrck krb5kdc[3399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422202, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:30:02.435395 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:02.427083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59436 (local address 127.0.98.62:33517)
0926 11:30:02.427257 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:02.427262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:02.427409 (+ 147us) server_negotiation.cc:407] Connection header received
0926 11:30:02.427607 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:02.427611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:02.427652 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:02.427735 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:02.428458 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:02.428976 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:02.429797 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:02.430017 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:02.430123 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:02.432363 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:02.432367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:02.432369 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:02.432400 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:02.433808 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:02.434245 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:02.434249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:02.434251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:02.434323 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:02.434568 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:02.434572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:02.434574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:02.434774 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:02.434913 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:02.435124 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 11:30:02.435234 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 11:30:02.435851 3444 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59436
I20260926 11:30:02.436836 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 3506
I20260926 11:30:02.443171 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 3640
I20260926 11:30:02.455832 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 3774
I20260926 11:30:02.461802 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 3412
2026-09-26T11:30:02Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2090 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:30:02 dist-test-slave-mrck krb5kdc[3915](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:30:02 dist-test-slave-mrck krb5kdc[3915](info): set up 2 sockets
Sep 26 11:30:02 dist-test-slave-mrck krb5kdc[3915](info): commencing operation
krb5kdc: starting...
W20260926 11:30:03.503800 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:30:03 dist-test-slave-mrck krb5kdc[3915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:30:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:30:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:30:03.626528 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44655
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44655
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:03.735307 3928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:03.735620 3928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:03.735690 3928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:03.739681 3928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:30:03.739820 3928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:03.739869 3928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:03.739907 3928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:30:03.739939 3928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:30:03.745858 3928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:03.746994 3928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:03.748088 3928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:03.756434 3934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:03.756454 3933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:03.756574 3928 server_base.cc:1034] running on GCE node
W20260926 11:30:03.756440 3936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:03.757232 3928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:03.758334 3928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:03.759490 3928 hybrid_clock.cc:648] HybridClock initialized: now 1790422203759477 us; error 28 us; skew 500 ppm
Sep 26 11:30:03 dist-test-slave-mrck krb5kdc[3915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422203, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:03.762269 3928 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:30:03.763118 3928 webserver.cc:460] Webserver started at http://127.0.98.62:35265/ using document root <none> and password file <none>
I20260926 11:30:03.763664 3928 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:03.763764 3928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:03.764007 3928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:03.766040 3928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "52b6047cbe284155995036ddcf86839e"
format_stamp: "Formatted at 2026-09-26 11:30:03 on dist-test-slave-mrck"
server_key: "c5e42dae9d9759d506a1b030580e37c8"
server_key_iv: "7ced1c05881d04ec709f79b25af4eb7c"
server_key_version: "encryptionkey@0"
I20260926 11:30:03.766592 3928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "52b6047cbe284155995036ddcf86839e"
format_stamp: "Formatted at 2026-09-26 11:30:03 on dist-test-slave-mrck"
server_key: "c5e42dae9d9759d506a1b030580e37c8"
server_key_iv: "7ced1c05881d04ec709f79b25af4eb7c"
server_key_version: "encryptionkey@0"
I20260926 11:30:03.770314 3928 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 11:30:03.772732 3943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:03.774003 3928 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:30:03.774125 3928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "52b6047cbe284155995036ddcf86839e"
format_stamp: "Formatted at 2026-09-26 11:30:03 on dist-test-slave-mrck"
server_key: "c5e42dae9d9759d506a1b030580e37c8"
server_key_iv: "7ced1c05881d04ec709f79b25af4eb7c"
server_key_version: "encryptionkey@0"
I20260926 11:30:03.774219 3928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:03.787161 3928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:03.790392 3928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:03.790629 3928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:03.797958 3928 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44655
I20260926 11:30:03.797972 3995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44655 every 8 connection(s)
I20260926 11:30:03.798952 3928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:30:03.801791 3996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:03.802825 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 3928
I20260926 11:30:03.802925 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:30:03.803231 392 external_mini_cluster.cc:1428] Setting key efce0784b7bd73ff2c8b9a1a72241de2
I20260926 11:30:03.807351 3996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52b6047cbe284155995036ddcf86839e: Bootstrap starting.
I20260926 11:30:03.809561 3996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52b6047cbe284155995036ddcf86839e: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:03.810276 3996 log.cc:824] T 00000000000000000000000000000000 P 52b6047cbe284155995036ddcf86839e: Log is configured to *not* fsync() on all Append() calls
Sep 26 11:30:03 dist-test-slave-mrck krb5kdc[3915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422203, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:30:03.812139 3996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52b6047cbe284155995036ddcf86839e: No bootstrap required, opened a new log
I20260926 11:30:03.815284 3996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52b6047cbe284155995036ddcf86839e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b6047cbe284155995036ddcf86839e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44655 } }
I20260926 11:30:03.815527 3996 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52b6047cbe284155995036ddcf86839e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:03.815634 3996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52b6047cbe284155995036ddcf86839e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52b6047cbe284155995036ddcf86839e, State: Initialized, Role: FOLLOWER
I20260926 11:30:03.816167 3996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52b6047cbe284155995036ddcf86839e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b6047cbe284155995036ddcf86839e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44655 } }
I20260926 11:30:03.816322 3996 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52b6047cbe284155995036ddcf86839e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:30:03.816402 3996 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52b6047cbe284155995036ddcf86839e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:30:03.816509 3996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52b6047cbe284155995036ddcf86839e [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:03.817610 3996 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52b6047cbe284155995036ddcf86839e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b6047cbe284155995036ddcf86839e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44655 } }
I20260926 11:30:03.817945 3996 leader_election.cc:302] T 00000000000000000000000000000000 P 52b6047cbe284155995036ddcf86839e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52b6047cbe284155995036ddcf86839e; no voters:
I20260926 11:30:03.818166 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:03.804628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52684 (local address 127.0.98.62:44655)
0926 11:30:03.805063 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:03.805079 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:03.805117 (+ 38us) server_negotiation.cc:407] Connection header received
0926 11:30:03.805778 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:03.805796 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:03.806110 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:03.806470 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:03.807394 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:03.808511 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:03.809423 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:03.809956 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:03.810127 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:03.812693 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:03.812698 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:03.812710 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:03.812765 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:03.814927 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:03.815355 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:03.815361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:03.815368 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:03.815480 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:03.815845 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:03.815848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:03.815850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:03.816333 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:03.816547 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:03.816855 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 11:30:03.817149 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":131,"threads_started":1}
I20260926 11:30:03.818259 3996 leader_election.cc:288] T 00000000000000000000000000000000 P 52b6047cbe284155995036ddcf86839e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:30:03.818423 4001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52b6047cbe284155995036ddcf86839e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:30:03.818651 4001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52b6047cbe284155995036ddcf86839e [term 1 LEADER]: Becoming Leader. State: Replica: 52b6047cbe284155995036ddcf86839e, State: Running, Role: LEADER
I20260926 11:30:03.819013 4001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52b6047cbe284155995036ddcf86839e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b6047cbe284155995036ddcf86839e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44655 } }
I20260926 11:30:03.819454 3996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52b6047cbe284155995036ddcf86839e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:30:03.820567 4003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52b6047cbe284155995036ddcf86839e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52b6047cbe284155995036ddcf86839e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b6047cbe284155995036ddcf86839e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44655 } } }
I20260926 11:30:03.820705 4003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52b6047cbe284155995036ddcf86839e [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:03.820922 4003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52b6047cbe284155995036ddcf86839e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52b6047cbe284155995036ddcf86839e. Latest consensus state: current_term: 1 leader_uuid: "52b6047cbe284155995036ddcf86839e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52b6047cbe284155995036ddcf86839e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44655 } } }
I20260926 11:30:03.821009 4003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52b6047cbe284155995036ddcf86839e [sys.catalog]: This master's current role is: LEADER
W20260926 11:30:03.824136 4016 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 52b6047cbe284155995036ddcf86839e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:30:03.824211 4016 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:30:03.824301 4010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:30:03.825026 4010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:30:03.829934 4010 catalog_manager.cc:1347] Generated new cluster ID: e1ad5a5b28e04420af585eba78e95bf9
I20260926 11:30:03.829996 4010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:30:03.863934 4010 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:30:03.865248 4010 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:30:03.870675 4010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52b6047cbe284155995036ddcf86839e: Generated new TSK 0
I20260926 11:30:03.871488 4010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:30:03.907979 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44655
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:04.016309 4022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:04.016583 4022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:04.016659 4022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:04.020859 4022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:04.020932 4022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:04.021019 4022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:30:04.026039 4022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:04.027046 4022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:04.028138 4022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:04.037426 4027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:04.037477 4028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:04.037448 4030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:04.037724 4022 server_base.cc:1034] running on GCE node
I20260926 11:30:04.038168 4022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:04.038774 4022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:04.039952 4022 hybrid_clock.cc:648] HybridClock initialized: now 1790422204039940 us; error 31 us; skew 500 ppm
Sep 26 11:30:04 dist-test-slave-mrck krb5kdc[3915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:04.042695 4022 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:30:04.043462 4022 webserver.cc:460] Webserver started at http://127.0.98.1:40953/ using document root <none> and password file <none>
I20260926 11:30:04.044039 4022 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:04.044096 4022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:04.044252 4022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:04.046131 4022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-0/data/instance:
uuid: "013cb74bd8624760a195784cb1ebea7a"
format_stamp: "Formatted at 2026-09-26 11:30:04 on dist-test-slave-mrck"
server_key: "5f2170a0d209e8e3fd757032e45e3458"
server_key_iv: "dffaec12dfc633134c1d78c41d49775a"
server_key_version: "encryptionkey@0"
I20260926 11:30:04.046633 4022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "013cb74bd8624760a195784cb1ebea7a"
format_stamp: "Formatted at 2026-09-26 11:30:04 on dist-test-slave-mrck"
server_key: "5f2170a0d209e8e3fd757032e45e3458"
server_key_iv: "dffaec12dfc633134c1d78c41d49775a"
server_key_version: "encryptionkey@0"
I20260926 11:30:04.050096 4022 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 11:30:04.052299 4037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:04.053756 4022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:30:04.053869 4022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "013cb74bd8624760a195784cb1ebea7a"
format_stamp: "Formatted at 2026-09-26 11:30:04 on dist-test-slave-mrck"
server_key: "5f2170a0d209e8e3fd757032e45e3458"
server_key_iv: "dffaec12dfc633134c1d78c41d49775a"
server_key_version: "encryptionkey@0"
I20260926 11:30:04.053979 4022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:04.074193 4022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:04.077284 4022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:04.077445 4022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:04.078027 4022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:04.078869 4022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:04.078922 4022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:04.078960 4022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:04.078982 4022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:04.087116 4022 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43005
I20260926 11:30:04.087185 4150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43005 every 8 connection(s)
I20260926 11:30:04.088106 4022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:30:04.094823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 4022
I20260926 11:30:04.094923 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:30:04.095223 392 external_mini_cluster.cc:1428] Setting key 750b5a8af823c2c9d75f5a18ce741e72
Sep 26 11:30:04 dist-test-slave-mrck krb5kdc[3915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:30:04.102573 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:04.089836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52415 (local address 127.0.98.62:44655)
0926 11:30:04.089979 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:04.089982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:04.090782 (+ 800us) server_negotiation.cc:407] Connection header received
0926 11:30:04.092182 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:04.092185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:04.092232 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:04.092311 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:04.093428 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:04.093956 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:04.095109 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:04.095416 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:04.095527 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:04.098362 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:04.098365 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:04.098367 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:04.098390 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:04.099973 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:04.100677 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:04.100683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:04.100687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:04.100821 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:04.101266 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:04.101270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:04.101273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:04.101583 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:04.101686 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:04.102341 (+ 655us) server_negotiation.cc:299] Negotiation successful
0926 11:30:04.102452 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 11:30:04.103828 4153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:04.090205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44655 (local address 127.0.98.1:52415)
0926 11:30:04.090626 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:04.090668 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:04.091970 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:04.092467 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:04.092473 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:04.092829 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:04.093293 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:04.093306 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:04.094060 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:04.094063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:04.094977 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:04.094995 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:04.095564 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:04.095567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:04.095793 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:04.096409 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:04.096429 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:04.098158 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:04.100084 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:04.100093 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:04.100111 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:04.100526 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:04.100913 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:04.100919 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:04.100924 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:04.101136 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:04.101710 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:04.101718 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:04.102068 (+ 350us) client_negotiation.cc:769] Sending connection context
0926 11:30:04.102304 (+ 236us) client_negotiation.cc:240] Negotiation successful
0926 11:30:04.102685 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 11:30:04.105219 4151 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44655
I20260926 11:30:04.105448 4151 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:04.105901 4151 heartbeater.cc:511] Master 127.0.98.62:44655 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 11:30:04.107517 3960 ts_manager.cc:194] Registered new tserver with Master: 013cb74bd8624760a195784cb1ebea7a (127.0.98.1:43005)
I20260926 11:30:04.109265 3960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52415
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:30:04.121888 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44655
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:04.238588 4156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:04.238852 4156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:04.238896 4156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:04.243067 4156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:04.243139 4156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:04.243201 4156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:30:04.248615 4156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:04.249691 4156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:04.250694 4156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:04.260226 4161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:04.260221 4164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:04.260221 4162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:04.260429 4156 server_base.cc:1034] running on GCE node
I20260926 11:30:04.260811 4156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:04.261351 4156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:04.262507 4156 hybrid_clock.cc:648] HybridClock initialized: now 1790422204262501 us; error 30 us; skew 500 ppm
Sep 26 11:30:04 dist-test-slave-mrck krb5kdc[3915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:04.265249 4156 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:30:04.266062 4156 webserver.cc:460] Webserver started at http://127.0.98.2:35263/ using document root <none> and password file <none>
I20260926 11:30:04.266561 4156 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:04.266616 4156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:04.266773 4156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:04.268800 4156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-1/data/instance:
uuid: "37a857ba04d44ee6991d25f0c6b4cbee"
format_stamp: "Formatted at 2026-09-26 11:30:04 on dist-test-slave-mrck"
server_key: "5b14da81831fb1e82337502354fb9cbf"
server_key_iv: "e74734fcbc4baf0b85b85f8f9bb54fd6"
server_key_version: "encryptionkey@0"
I20260926 11:30:04.269306 4156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "37a857ba04d44ee6991d25f0c6b4cbee"
format_stamp: "Formatted at 2026-09-26 11:30:04 on dist-test-slave-mrck"
server_key: "5b14da81831fb1e82337502354fb9cbf"
server_key_iv: "e74734fcbc4baf0b85b85f8f9bb54fd6"
server_key_version: "encryptionkey@0"
I20260926 11:30:04.272805 4156 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 11:30:04.274992 4171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:04.276175 4156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:30:04.276311 4156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "37a857ba04d44ee6991d25f0c6b4cbee"
format_stamp: "Formatted at 2026-09-26 11:30:04 on dist-test-slave-mrck"
server_key: "5b14da81831fb1e82337502354fb9cbf"
server_key_iv: "e74734fcbc4baf0b85b85f8f9bb54fd6"
server_key_version: "encryptionkey@0"
I20260926 11:30:04.276417 4156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:04.292733 4156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:04.295584 4156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:04.295859 4156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:04.296450 4156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:04.297262 4156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:04.297333 4156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:04.297400 4156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:04.297451 4156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:04.306300 4156 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46467
I20260926 11:30:04.306329 4284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46467 every 8 connection(s)
I20260926 11:30:04.307206 4156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:30:04.308275 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 4156
I20260926 11:30:04.308404 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:30:04.308712 392 external_mini_cluster.cc:1428] Setting key 713ef0aba9359bc2091d7a097ed1b695
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 26 11:30:04 dist-test-slave-mrck krb5kdc[3915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 11:30:04.322898 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:04.309204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58119 (local address 127.0.98.62:44655)
0926 11:30:04.309337 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:04.309341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:04.310406 (+ 1065us) server_negotiation.cc:407] Connection header received
0926 11:30:04.312391 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:04.312397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:04.312458 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:04.312557 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:04.313680 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:04.314595 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:04.315709 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:04.316152 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:04.316273 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:04.318881 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:04.318885 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:04.318887 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:04.318928 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:04.320458 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:04.321076 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:04.321083 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:04.321088 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:04.321204 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:04.321634 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:04.321640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:04.321645 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:04.321910 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:04.322025 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:04.322670 (+ 645us) server_negotiation.cc:299] Negotiation successful
0926 11:30:04.322775 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 11:30:04.323922 4287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:04.309672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44655 (local address 127.0.98.2:58119)
0926 11:30:04.310215 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:04.310272 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:04.312159 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:04.312712 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:04.312718 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:04.313068 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:04.313535 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:04.313547 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:04.314717 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:04.314720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:04.315584 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:04.315593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:04.316298 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:04.316301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:04.316458 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:04.317069 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:04.317091 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:04.318648 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:04.320574 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:04.320602 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:04.320621 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:04.320963 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:04.321335 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:04.321338 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:04.321340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:04.321533 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:04.322037 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:04.322046 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:04.322404 (+ 358us) client_negotiation.cc:769] Sending connection context
0926 11:30:04.322632 (+ 228us) client_negotiation.cc:240] Negotiation successful
0926 11:30:04.323009 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":144,"threads_started":1}
I20260926 11:30:04.325285 4285 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44655
I20260926 11:30:04.325539 4285 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:04.326053 4285 heartbeater.cc:511] Master 127.0.98.62:44655 requested a full tablet report, sending...
I20260926 11:30:04.327145 3960 ts_manager.cc:194] Registered new tserver with Master: 37a857ba04d44ee6991d25f0c6b4cbee (127.0.98.2:46467)
I20260926 11:30:04.327709 3960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58119
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:30:04.336936 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44655
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:04.446293 4290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:04.446549 4290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:04.446588 4290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:04.450551 4290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:04.450608 4290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:04.450662 4290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:30:04.455770 4290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44655
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:04.456782 4290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:04.457801 4290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:04.467135 4295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:04.467311 4290 server_base.cc:1034] running on GCE node
W20260926 11:30:04.467216 4298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:04.467135 4296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:04.467911 4290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:04.468533 4290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:04.469713 4290 hybrid_clock.cc:648] HybridClock initialized: now 1790422204469695 us; error 39 us; skew 500 ppm
Sep 26 11:30:04 dist-test-slave-mrck krb5kdc[3915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:04.472424 4290 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:30:04.473245 4290 webserver.cc:460] Webserver started at http://127.0.98.3:37895/ using document root <none> and password file <none>
I20260926 11:30:04.473773 4290 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:04.473827 4290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:04.473987 4290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:04.475907 4290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-2/data/instance:
uuid: "683b40eaf58b4cafa0b7940de4f36fdd"
format_stamp: "Formatted at 2026-09-26 11:30:04 on dist-test-slave-mrck"
server_key: "e8af09c6801fa97030776aa1243cf2a0"
server_key_iv: "39d644339f317c6299e71f25c7613237"
server_key_version: "encryptionkey@0"
I20260926 11:30:04.476385 4290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "683b40eaf58b4cafa0b7940de4f36fdd"
format_stamp: "Formatted at 2026-09-26 11:30:04 on dist-test-slave-mrck"
server_key: "e8af09c6801fa97030776aa1243cf2a0"
server_key_iv: "39d644339f317c6299e71f25c7613237"
server_key_version: "encryptionkey@0"
I20260926 11:30:04.480340 4290 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 11:30:04.482720 4305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:04.484046 4290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:30:04.484138 4290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "683b40eaf58b4cafa0b7940de4f36fdd"
format_stamp: "Formatted at 2026-09-26 11:30:04 on dist-test-slave-mrck"
server_key: "e8af09c6801fa97030776aa1243cf2a0"
server_key_iv: "39d644339f317c6299e71f25c7613237"
server_key_version: "encryptionkey@0"
I20260926 11:30:04.484222 4290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:04.501281 4290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:04.504627 4290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:04.504791 4290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:04.505316 4290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:04.506065 4290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:04.506109 4290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:04.506146 4290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:04.506165 4290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:04.514961 4290 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42299
I20260926 11:30:04.514979 4418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42299 every 8 connection(s)
I20260926 11:30:04.515928 4290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:30:04.523106 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 4290
I20260926 11:30:04.523240 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790422180210734-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:30:04.523578 392 external_mini_cluster.cc:1428] Setting key c28523ecaa35835a1a5d408b0e16d88a
Sep 26 11:30:04 dist-test-slave-mrck krb5kdc[3915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422204, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:30:04.529724 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:04.517702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40445 (local address 127.0.98.62:44655)
0926 11:30:04.517848 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:04.517852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:04.518643 (+ 791us) server_negotiation.cc:407] Connection header received
0926 11:30:04.520205 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:04.520209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:04.520268 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:04.520383 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:04.521676 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:04.522183 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:04.523308 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:04.523640 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:04.523777 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:04.526236 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:04.526248 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:04.526251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:04.526274 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:04.527694 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:04.528270 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:04.528275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:04.528277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:04.528358 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:04.528704 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:04.528707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:04.528709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:04.528894 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:04.529001 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:04.529472 (+ 471us) server_negotiation.cc:299] Negotiation successful
0926 11:30:04.529576 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 11:30:04.530494 4421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:04.518075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44655 (local address 127.0.98.3:40445)
0926 11:30:04.518496 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:04.518537 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:04.519992 (+ 1455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:04.520578 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:04.520586 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:04.520994 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:04.521517 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:04.521531 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:04.522302 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:04.522305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:04.523155 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:04.523164 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:04.523827 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:04.523830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:04.523974 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:04.524662 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:04.524678 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:04.526080 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:04.527872 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:04.527878 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:04.527893 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:04.528162 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:04.528455 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:04.528458 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:04.528460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:04.528603 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:04.528999 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:04.529006 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:04.529247 (+ 241us) client_negotiation.cc:769] Sending connection context
0926 11:30:04.529436 (+ 189us) client_negotiation.cc:240] Negotiation successful
0926 11:30:04.529740 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":103,"threads_started":1}
I20260926 11:30:04.531466 4419 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44655
I20260926 11:30:04.531708 4419 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:04.532212 4419 heartbeater.cc:511] Master 127.0.98.62:44655 requested a full tablet report, sending...
I20260926 11:30:04.533201 3960 ts_manager.cc:194] Registered new tserver with Master: 683b40eaf58b4cafa0b7940de4f36fdd (127.0.98.3:42299)
I20260926 11:30:04.533691 3960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40445
I20260926 11:30:04.537189 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 11:30:04 dist-test-slave-mrck krb5kdc[3915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 11:30:04 dist-test-slave-mrck krb5kdc[3915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 11:30:04.562580 4426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:04.553030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48318 (local address 127.0.98.1:43005)
0926 11:30:04.553344 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:04.553360 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:04.553403 (+ 43us) server_negotiation.cc:407] Connection header received
0926 11:30:04.553469 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:04.553474 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:04.553670 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:04.553819 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:04.554495 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:04.555260 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:04.556131 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:04.556499 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:04.556615 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:04.559255 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:04.559263 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:04.559275 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:04.559297 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:04.560781 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:04.561216 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:04.561221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:04.561223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:04.561300 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:04.561572 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:04.561577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:04.561578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:04.561809 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:04.562068 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:04.562233 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 11:30:04.562400 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":113,"threads_started":1}
Sep 26 11:30:04 dist-test-slave-mrck krb5kdc[3915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:30:04.573493 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:04.565442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52686 (local address 127.0.98.62:44655)
0926 11:30:04.565591 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:04.565595 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:04.565609 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:30:04.565711 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:04.565714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:04.565756 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:04.565835 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:04.566410 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:04.566930 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:04.567639 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:04.567923 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:04.568028 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:04.570238 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:04.570243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:04.570246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:04.570280 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:04.571834 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:04.572197 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:04.572200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:04.572201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:04.572306 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:04.572612 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:04.572616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:04.572617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:04.572843 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:04.572965 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:04.573154 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 11:30:04.573253 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260926 11:30:04.573926 3960 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52686
I20260926 11:30:04.574605 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 4022
I20260926 11:30:04.580467 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 4156
I20260926 11:30:04.586119 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 4290
I20260926 11:30:04.591569 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 3928
2026-09-26T11:30:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2128 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:30:04 dist-test-slave-mrck krb5kdc[4431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:30:04 dist-test-slave-mrck krb5kdc[4431](info): set up 2 sockets
Sep 26 11:30:04 dist-test-slave-mrck krb5kdc[4431](info): commencing operation
krb5kdc: starting...
W20260926 11:30:05.619800 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:30:05 dist-test-slave-mrck krb5kdc[4431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:30:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:30:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:30:05.735098 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44743
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:05.841885 4444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:05.842151 4444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:05.842195 4444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:05.846088 4444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:30:05.846141 4444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:05.846159 4444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:05.846194 4444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:30:05.846210 4444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:30:05.851384 4444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:05.852387 4444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:05.853339 4444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:05.861589 4449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:05.861570 4452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:05.861666 4450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:05.861927 4444 server_base.cc:1034] running on GCE node
I20260926 11:30:05.862473 4444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:05.863545 4444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:05.864711 4444 hybrid_clock.cc:648] HybridClock initialized: now 1790422205864686 us; error 48 us; skew 500 ppm
Sep 26 11:30:05 dist-test-slave-mrck krb5kdc[4431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422205, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:05.871127 4444 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:30:05.872020 4444 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:05.872123 4444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:05.872354 4444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:05.874394 4444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "a100619781b344abbd835061961e80cd"
format_stamp: "Formatted at 2026-09-26 11:30:05 on dist-test-slave-mrck"
server_key: "0f00383653212572bda2fdb2708e6aae"
server_key_iv: "976a3bf4807b82a88acd0785f1ae2862"
server_key_version: "encryptionkey@0"
I20260926 11:30:05.874962 4444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "a100619781b344abbd835061961e80cd"
format_stamp: "Formatted at 2026-09-26 11:30:05 on dist-test-slave-mrck"
server_key: "0f00383653212572bda2fdb2708e6aae"
server_key_iv: "976a3bf4807b82a88acd0785f1ae2862"
server_key_version: "encryptionkey@0"
I20260926 11:30:05.878726 4444 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 11:30:05.881138 4458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:05.882346 4444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 11:30:05.882500 4444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "a100619781b344abbd835061961e80cd"
format_stamp: "Formatted at 2026-09-26 11:30:05 on dist-test-slave-mrck"
server_key: "0f00383653212572bda2fdb2708e6aae"
server_key_iv: "976a3bf4807b82a88acd0785f1ae2862"
server_key_version: "encryptionkey@0"
I20260926 11:30:05.882611 4444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:05.914470 4444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:05.917461 4444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:05.917680 4444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:05.924770 4444 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44743
I20260926 11:30:05.925073 4510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44743 every 8 connection(s)
I20260926 11:30:05.925472 4444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:30:05.928218 4511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:05.931886 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 4444
I20260926 11:30:05.932039 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:30:05.932299 392 external_mini_cluster.cc:1428] Setting key 252a121c790b0f589788d7985aa44084
I20260926 11:30:05.933961 4511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a100619781b344abbd835061961e80cd: Bootstrap starting.
I20260926 11:30:05.936476 4511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a100619781b344abbd835061961e80cd: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:05.937198 4511 log.cc:824] T 00000000000000000000000000000000 P a100619781b344abbd835061961e80cd: Log is configured to *not* fsync() on all Append() calls
I20260926 11:30:05.939280 4511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a100619781b344abbd835061961e80cd: No bootstrap required, opened a new log
Sep 26 11:30:05 dist-test-slave-mrck krb5kdc[4431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:30:05.942723 4511 raft_consensus.cc:348] T 00000000000000000000000000000000 P a100619781b344abbd835061961e80cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a100619781b344abbd835061961e80cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44743 } }
I20260926 11:30:05.942931 4511 raft_consensus.cc:374] T 00000000000000000000000000000000 P a100619781b344abbd835061961e80cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:05.942999 4511 raft_consensus.cc:729] T 00000000000000000000000000000000 P a100619781b344abbd835061961e80cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a100619781b344abbd835061961e80cd, State: Initialized, Role: FOLLOWER
I20260926 11:30:05.943450 4511 consensus_queue.cc:260] T 00000000000000000000000000000000 P a100619781b344abbd835061961e80cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a100619781b344abbd835061961e80cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44743 } }
I20260926 11:30:05.943568 4511 raft_consensus.cc:388] T 00000000000000000000000000000000 P a100619781b344abbd835061961e80cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:30:05.943645 4511 raft_consensus.cc:482] T 00000000000000000000000000000000 P a100619781b344abbd835061961e80cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:30:05.943784 4511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a100619781b344abbd835061961e80cd [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:05.944621 4511 raft_consensus.cc:504] T 00000000000000000000000000000000 P a100619781b344abbd835061961e80cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a100619781b344abbd835061961e80cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44743 } }
I20260926 11:30:05.944962 4511 leader_election.cc:302] T 00000000000000000000000000000000 P a100619781b344abbd835061961e80cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a100619781b344abbd835061961e80cd; no voters:
I20260926 11:30:05.945259 4511 leader_election.cc:288] T 00000000000000000000000000000000 P a100619781b344abbd835061961e80cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:30:05.945361 4516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a100619781b344abbd835061961e80cd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:30:05.945572 4516 raft_consensus.cc:686] T 00000000000000000000000000000000 P a100619781b344abbd835061961e80cd [term 1 LEADER]: Becoming Leader. State: Replica: a100619781b344abbd835061961e80cd, State: Running, Role: LEADER
I20260926 11:30:05.946461 4516 consensus_queue.cc:237] T 00000000000000000000000000000000 P a100619781b344abbd835061961e80cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a100619781b344abbd835061961e80cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44743 } }
I20260926 11:30:05.946566 4511 sys_catalog.cc:564] T 00000000000000000000000000000000 P a100619781b344abbd835061961e80cd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:30:05.947345 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:05.933717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52828 (local address 127.0.98.62:44743)
0926 11:30:05.934108 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:05.934124 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:05.934159 (+ 35us) server_negotiation.cc:407] Connection header received
0926 11:30:05.934764 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:05.934779 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:05.935116 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:05.935393 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:05.936312 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:05.937378 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:05.938447 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:05.938900 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:05.939013 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:05.941796 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:05.941801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:05.941813 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:05.941836 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:05.944206 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:05.944633 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:05.944639 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:05.944644 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:05.944810 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:05.945106 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:05.945109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:05.945111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:05.945627 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:05.945815 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:05.946080 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 11:30:05.946358 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":130,"threads_started":1}
I20260926 11:30:05.949182 4518 sys_catalog.cc:455] T 00000000000000000000000000000000 P a100619781b344abbd835061961e80cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a100619781b344abbd835061961e80cd. Latest consensus state: current_term: 1 leader_uuid: "a100619781b344abbd835061961e80cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a100619781b344abbd835061961e80cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44743 } } }
I20260926 11:30:05.949298 4518 sys_catalog.cc:458] T 00000000000000000000000000000000 P a100619781b344abbd835061961e80cd [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:05.950650 4517 sys_catalog.cc:455] T 00000000000000000000000000000000 P a100619781b344abbd835061961e80cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a100619781b344abbd835061961e80cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a100619781b344abbd835061961e80cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44743 } } }
I20260926 11:30:05.950785 4517 sys_catalog.cc:458] T 00000000000000000000000000000000 P a100619781b344abbd835061961e80cd [sys.catalog]: This master's current role is: LEADER
W20260926 11:30:05.952013 4530 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a100619781b344abbd835061961e80cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:30:05.952105 4530 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:30:05.952179 4532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:30:05.953018 4532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:30:05.958511 4532 catalog_manager.cc:1347] Generated new cluster ID: 31969eed30c24056b79c4939fc202d72
I20260926 11:30:05.958586 4532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:30:05.969794 4532 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:30:05.970727 4532 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:30:05.977206 4532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a100619781b344abbd835061961e80cd: Generated new TSK 0
I20260926 11:30:05.977988 4532 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:30:06.009203 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44743
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:06.119940 4537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:06.120237 4537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:06.120319 4537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:06.124398 4537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:06.124483 4537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:06.124575 4537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:30:06.129731 4537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:06.130821 4537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:06.131922 4537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:06.140949 4545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:06.140974 4542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:06.140949 4543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:06.141290 4537 server_base.cc:1034] running on GCE node
I20260926 11:30:06.141721 4537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:06.142338 4537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:06.143507 4537 hybrid_clock.cc:648] HybridClock initialized: now 1790422206143490 us; error 28 us; skew 500 ppm
Sep 26 11:30:06 dist-test-slave-mrck krb5kdc[4431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:06.146432 4537 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:30:06.147189 4537 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:06.147253 4537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:06.147457 4537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:06.149716 4537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-0/data/instance:
uuid: "2cc5da2e62d84a2db3307053d9460b4f"
format_stamp: "Formatted at 2026-09-26 11:30:06 on dist-test-slave-mrck"
server_key: "7a71988d06f29ee363fa84d0dda6fdd8"
server_key_iv: "9996ed84c55cb48f25501505f7cda94c"
server_key_version: "encryptionkey@0"
I20260926 11:30:06.150220 4537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2cc5da2e62d84a2db3307053d9460b4f"
format_stamp: "Formatted at 2026-09-26 11:30:06 on dist-test-slave-mrck"
server_key: "7a71988d06f29ee363fa84d0dda6fdd8"
server_key_iv: "9996ed84c55cb48f25501505f7cda94c"
server_key_version: "encryptionkey@0"
I20260926 11:30:06.153785 4537 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 11:30:06.156134 4551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:06.157334 4537 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:30:06.157426 4537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "2cc5da2e62d84a2db3307053d9460b4f"
format_stamp: "Formatted at 2026-09-26 11:30:06 on dist-test-slave-mrck"
server_key: "7a71988d06f29ee363fa84d0dda6fdd8"
server_key_iv: "9996ed84c55cb48f25501505f7cda94c"
server_key_version: "encryptionkey@0"
I20260926 11:30:06.157509 4537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:06.181169 4537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:06.184438 4537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:06.184602 4537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:06.185115 4537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:06.185945 4537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:06.185989 4537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:06.186046 4537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:06.186062 4537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:06.194679 4537 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38207
I20260926 11:30:06.194696 4664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38207 every 8 connection(s)
I20260926 11:30:06.195253 4537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:30:06.195852 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 4537
I20260926 11:30:06.195986 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:30:06.196259 392 external_mini_cluster.cc:1428] Setting key 505bb2a72cd8b4c949d0aefaf78cd7f2
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 26 11:30:06 dist-test-slave-mrck krb5kdc[4431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:30:06.213732 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:06.197097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57813 (local address 127.0.98.62:44743)
0926 11:30:06.197225 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:06.197229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:06.198319 (+ 1090us) server_negotiation.cc:407] Connection header received
0926 11:30:06.199955 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:06.199958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:06.200003 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:06.200088 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:06.201735 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.202264 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.204157 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.204411 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.204504 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:06.209982 (+ 5478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:06.209985 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:06.209987 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:06.210015 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:06.211559 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:06.212144 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:06.212152 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:06.212153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:06.212229 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:06.212591 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:06.212594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:06.212597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:06.212787 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:06.212894 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:06.213423 (+ 529us) server_negotiation.cc:299] Negotiation successful
0926 11:30:06.213554 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 11:30:06.214561 4667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:06.197661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44743 (local address 127.0.98.1:57813)
0926 11:30:06.198182 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:06.198225 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:06.199682 (+ 1457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:06.200279 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:06.200305 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:06.200854 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:06.201554 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:06.201578 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.202395 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.202399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:06.204012 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:06.204025 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.204583 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.204587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:06.204815 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:06.205502 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:06.205525 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:06.209759 (+ 4234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:06.211692 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:06.211698 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:06.211717 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:06.212026 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:06.212323 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:06.212326 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:06.212328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:06.212488 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:06.212894 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:06.212902 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:06.213171 (+ 269us) client_negotiation.cc:769] Sending connection context
0926 11:30:06.213362 (+ 191us) client_negotiation.cc:240] Negotiation successful
0926 11:30:06.213703 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":161,"threads_started":1}
I20260926 11:30:06.215904 4665 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44743
I20260926 11:30:06.216152 4665 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:06.216607 4665 heartbeater.cc:511] Master 127.0.98.62:44743 requested a full tablet report, sending...
I20260926 11:30:06.218223 4475 ts_manager.cc:194] Registered new tserver with Master: 2cc5da2e62d84a2db3307053d9460b4f (127.0.98.1:38207)
I20260926 11:30:06.219374 4475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:30:06.222707 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44743
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:06.332660 4670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:06.332957 4670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:06.333006 4670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:06.336982 4670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:06.337037 4670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:06.337163 4670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:30:06.342594 4670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:06.343618 4670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:06.344666 4670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:06.354905 4675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:06.354907 4678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:06.355011 4670 server_base.cc:1034] running on GCE node
W20260926 11:30:06.354905 4676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:06.355844 4670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:06.356509 4670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:06.357720 4670 hybrid_clock.cc:648] HybridClock initialized: now 1790422206357698 us; error 43 us; skew 500 ppm
Sep 26 11:30:06 dist-test-slave-mrck krb5kdc[4431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:06.360576 4670 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:30:06.361368 4670 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:06.361439 4670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:06.361608 4670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:06.363605 4670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-1/data/instance:
uuid: "4f1b6a1a2adc4f4dbfb8f1c9b03ab658"
format_stamp: "Formatted at 2026-09-26 11:30:06 on dist-test-slave-mrck"
server_key: "901e7661afa9da6e0b66ebabd340b955"
server_key_iv: "c3c7b5b521768a113b391156c26fd946"
server_key_version: "encryptionkey@0"
I20260926 11:30:06.364188 4670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4f1b6a1a2adc4f4dbfb8f1c9b03ab658"
format_stamp: "Formatted at 2026-09-26 11:30:06 on dist-test-slave-mrck"
server_key: "901e7661afa9da6e0b66ebabd340b955"
server_key_iv: "c3c7b5b521768a113b391156c26fd946"
server_key_version: "encryptionkey@0"
I20260926 11:30:06.367676 4670 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:30:06.370100 4684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:06.371284 4670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 11:30:06.371431 4670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "4f1b6a1a2adc4f4dbfb8f1c9b03ab658"
format_stamp: "Formatted at 2026-09-26 11:30:06 on dist-test-slave-mrck"
server_key: "901e7661afa9da6e0b66ebabd340b955"
server_key_iv: "c3c7b5b521768a113b391156c26fd946"
server_key_version: "encryptionkey@0"
I20260926 11:30:06.371536 4670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:06.401551 4670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:06.404850 4670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:06.405057 4670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:06.405673 4670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:06.406517 4670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:06.406562 4670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:06.406631 4670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:06.406672 4670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:06.415817 4670 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39631
I20260926 11:30:06.415838 4797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39631 every 8 connection(s)
I20260926 11:30:06.416442 4670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:30:06.419039 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 4670
I20260926 11:30:06.419186 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:30:06.419471 392 external_mini_cluster.cc:1428] Setting key ba345c4b8583f044214cc181f96a937f
Sep 26 11:30:06 dist-test-slave-mrck krb5kdc[4431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 11:30:06.432236 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:06.418245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58005 (local address 127.0.98.62:44743)
0926 11:30:06.418413 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:06.418418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:06.419363 (+ 945us) server_negotiation.cc:407] Connection header received
0926 11:30:06.421073 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:06.421076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:06.421129 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:06.421207 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:06.422510 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.423035 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.424922 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.425151 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.425245 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:06.428348 (+ 3103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:06.428352 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:06.428355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:06.428379 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:06.429827 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:06.430398 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:06.430404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:06.430409 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:06.430501 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:06.430863 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:06.430866 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:06.430867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:06.431053 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:06.431132 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:06.431706 (+ 574us) server_negotiation.cc:299] Negotiation successful
0926 11:30:06.431957 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 11:30:06.432925 4800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:06.418712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44743 (local address 127.0.98.2:58005)
0926 11:30:06.419206 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:06.419259 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:06.420828 (+ 1569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:06.421386 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:06.421394 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:06.421757 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:06.422343 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:06.422365 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.423166 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.423170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:06.424762 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:06.424776 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.425323 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.425330 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:06.425575 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:06.426231 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:06.426256 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:06.428169 (+ 1913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:06.429955 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:06.429960 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:06.429974 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:06.430224 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:06.430611 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:06.430613 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:06.430615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:06.430750 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:06.431145 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:06.431150 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:06.431442 (+ 292us) client_negotiation.cc:769] Sending connection context
0926 11:30:06.431597 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 11:30:06.431973 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":134,"threads_started":1}
I20260926 11:30:06.434216 4798 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44743
I20260926 11:30:06.434484 4798 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:06.434950 4798 heartbeater.cc:511] Master 127.0.98.62:44743 requested a full tablet report, sending...
I20260926 11:30:06.436089 4475 ts_manager.cc:194] Registered new tserver with Master: 4f1b6a1a2adc4f4dbfb8f1c9b03ab658 (127.0.98.2:39631)
I20260926 11:30:06.436693 4475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58005
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:30:06.447081 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44743
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:06.554515 4803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:06.554797 4803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:06.554885 4803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:06.558869 4803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:06.558951 4803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:06.559037 4803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:30:06.564328 4803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:06.565339 4803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:06.566298 4803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:06.575362 4809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:06.575414 4811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:06.575383 4808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:06.575872 4803 server_base.cc:1034] running on GCE node
I20260926 11:30:06.576220 4803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:06.576776 4803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:06.577975 4803 hybrid_clock.cc:648] HybridClock initialized: now 1790422206577959 us; error 27 us; skew 500 ppm
Sep 26 11:30:06 dist-test-slave-mrck krb5kdc[4431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:06.580946 4803 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:30:06.581766 4803 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:06.581830 4803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:06.582051 4803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:06.584121 4803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-2/data/instance:
uuid: "3e2eccf86bf34919aeb249bac4dca518"
format_stamp: "Formatted at 2026-09-26 11:30:06 on dist-test-slave-mrck"
server_key: "8acdfe0772a6ab20618f0c5614d7edba"
server_key_iv: "8d94b002b754209481d0ae3c87b559d9"
server_key_version: "encryptionkey@0"
I20260926 11:30:06.584664 4803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3e2eccf86bf34919aeb249bac4dca518"
format_stamp: "Formatted at 2026-09-26 11:30:06 on dist-test-slave-mrck"
server_key: "8acdfe0772a6ab20618f0c5614d7edba"
server_key_iv: "8d94b002b754209481d0ae3c87b559d9"
server_key_version: "encryptionkey@0"
I20260926 11:30:06.588347 4803 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:30:06.590636 4817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:06.591907 4803 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:30:06.592036 4803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "3e2eccf86bf34919aeb249bac4dca518"
format_stamp: "Formatted at 2026-09-26 11:30:06 on dist-test-slave-mrck"
server_key: "8acdfe0772a6ab20618f0c5614d7edba"
server_key_iv: "8d94b002b754209481d0ae3c87b559d9"
server_key_version: "encryptionkey@0"
I20260926 11:30:06.592144 4803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:06.609972 4803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:06.612907 4803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:06.613098 4803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:06.613699 4803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:06.614467 4803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:06.614530 4803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:06.614594 4803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:06.614640 4803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:06.623150 4803 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44787
I20260926 11:30:06.623174 4930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44787 every 8 connection(s)
I20260926 11:30:06.623713 4803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 11:30:06 dist-test-slave-mrck krb5kdc[4431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:30:06.633394 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 4803
I20260926 11:30:06.633533 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:30:06.633829 392 external_mini_cluster.cc:1428] Setting key a0e7d42d588c810a4ba5267c3efdc790
I20260926 11:30:06.637491 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:06.625514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54453 (local address 127.0.98.62:44743)
0926 11:30:06.625690 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:06.625694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:06.626512 (+ 818us) server_negotiation.cc:407] Connection header received
0926 11:30:06.627924 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:06.627927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:06.628004 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:06.628114 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:06.629300 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.629834 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.630995 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.631220 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.631341 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:06.633787 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:06.633793 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:06.633800 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:06.633826 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:06.635480 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:06.636061 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:06.636066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:06.636068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:06.636155 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:06.636543 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:06.636545 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:06.636546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:06.636715 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:06.636786 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:06.637269 (+ 483us) server_negotiation.cc:299] Negotiation successful
0926 11:30:06.637378 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 11:30:06.638376 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:06.625886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44743 (local address 127.0.98.3:54453)
0926 11:30:06.626361 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:06.626404 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:06.627646 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:06.628290 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:06.628297 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:06.628655 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:06.629134 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:06.629161 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.629981 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.629986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:06.630867 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:06.630876 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.631369 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.631373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:06.631553 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:06.632135 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:06.632150 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:06.633628 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:06.635593 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:06.635599 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:06.635613 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:06.635945 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:06.636273 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:06.636276 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:06.636278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:06.636449 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:06.636809 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:06.636814 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:06.637042 (+ 228us) client_negotiation.cc:769] Sending connection context
0926 11:30:06.637221 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 11:30:06.637480 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":109,"threads_started":1}
I20260926 11:30:06.639442 4931 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44743
I20260926 11:30:06.639654 4931 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:06.640079 4931 heartbeater.cc:511] Master 127.0.98.62:44743 requested a full tablet report, sending...
I20260926 11:30:06.640983 4475 ts_manager.cc:194] Registered new tserver with Master: 3e2eccf86bf34919aeb249bac4dca518 (127.0.98.3:44787)
I20260926 11:30:06.641538 4475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54453
I20260926 11:30:06.648101 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:30:06.657181 4514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:06.650001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52832 (local address 127.0.98.62:44743)
0926 11:30:06.650112 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:06.650117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:06.650310 (+ 193us) server_negotiation.cc:407] Connection header received
0926 11:30:06.650413 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:06.650415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:06.650450 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:06.650514 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:06.651179 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.651682 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.652566 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.652767 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.652840 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:06.654104 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:06.654107 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:06.654109 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:06.654129 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:06.655577 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:06.656137 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:06.656140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:06.656141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:06.656214 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:06.656452 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:06.656454 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:06.656455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:06.656659 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:06.656769 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:06.656979 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 11:30:06.657066 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 11:30:06.659879 4475 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:30:06.663043 4475 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:30:06.676988 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:06.671085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44787 (local address 127.0.0.1:49364)
0926 11:30:06.671580 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:06.671625 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:06.671801 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:06.672674 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:06.672682 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:06.672709 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:30:06.672831 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:06.672837 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.674077 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.674084 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:06.675389 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:06.675399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.676444 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.676448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:06.676547 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:06.676587 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 11:30:06.676702 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 11:30:06.676816 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":56,"thread_start_us":82,"threads_started":1}
I20260926 11:30:06.677001 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:06.671270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49364 (local address 127.0.98.3:44787)
0926 11:30:06.671968 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:06.671978 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:06.671997 (+ 19us) server_negotiation.cc:407] Connection header received
0926 11:30:06.672088 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:06.672093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:06.672309 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:06.672474 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:30:06.672954 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.673932 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.675520 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.676332 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.676432 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:06.676562 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:06.676747 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 11:30:06.676837 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":155,"threads_started":1}
I20260926 11:30:06.677510 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:06.671952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39631 (local address 127.0.0.1:32794)
0926 11:30:06.672548 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:06.672561 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:06.672640 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:06.673333 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:06.673336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:06.673352 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:30:06.673441 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:06.673449 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.674808 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.674811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:06.676017 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:06.676029 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.677108 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.677113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:06.677208 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:06.677228 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 11:30:06.677297 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 11:30:06.677359 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":20,"thread_start_us":100,"threads_started":1}
I20260926 11:30:06.677595 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:06.672165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32794 (local address 127.0.98.2:39631)
0926 11:30:06.672864 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:06.672891 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:06.672914 (+ 23us) server_negotiation.cc:407] Connection header received
0926 11:30:06.672987 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:06.672991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:06.673167 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:06.673301 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:30:06.673556 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.674676 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.676142 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.676827 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.676920 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:06.677041 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:06.677352 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 11:30:06.677443 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":126,"threads_started":1}
I20260926 11:30:06.677824 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:06.672330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35680 (local address 127.0.98.1:38207)
0926 11:30:06.672686 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:06.672694 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:06.672724 (+ 30us) server_negotiation.cc:407] Connection header received
0926 11:30:06.672797 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:06.672802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:06.673009 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:06.673164 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:30:06.673432 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.674609 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.676173 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.676912 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.677008 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:06.677142 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:06.677568 (+ 426us) server_negotiation.cc:299] Negotiation successful
0926 11:30:06.677666 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":87,"threads_started":1}
I20260926 11:30:06.678059 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:06.671289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38207 (local address 127.0.0.1:35680)
0926 11:30:06.671683 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:06.671698 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:06.671807 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:06.673159 (+ 1352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:06.673162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:06.673179 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:30:06.673292 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:06.673298 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.674736 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.674739 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:06.676041 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:06.676051 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.677202 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.677206 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:06.677312 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:06.677334 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 11:30:06.677402 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 11:30:06.677464 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":2,"thread_start_us":114,"threads_started":1}
I20260926 11:30:06.680298 4732 tablet_service.cc:1467] Processing CreateTablet for tablet 068ab849591a4ffda38ea559b0612cf1 (DEFAULT_TABLE table=test-table [id=e2e02e9938554e498d78aa8350921324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:30:06.680305 4865 tablet_service.cc:1467] Processing CreateTablet for tablet 068ab849591a4ffda38ea559b0612cf1 (DEFAULT_TABLE table=test-table [id=e2e02e9938554e498d78aa8350921324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:30:06.680709 4599 tablet_service.cc:1467] Processing CreateTablet for tablet 068ab849591a4ffda38ea559b0612cf1 (DEFAULT_TABLE table=test-table [id=e2e02e9938554e498d78aa8350921324]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:30:06.681278 4732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 068ab849591a4ffda38ea559b0612cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:06.681294 4865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 068ab849591a4ffda38ea559b0612cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:06.681566 4599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 068ab849591a4ffda38ea559b0612cf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:06.687052 4948 tablet_bootstrap.cc:492] T 068ab849591a4ffda38ea559b0612cf1 P 4f1b6a1a2adc4f4dbfb8f1c9b03ab658: Bootstrap starting.
I20260926 11:30:06.688575 4950 tablet_bootstrap.cc:492] T 068ab849591a4ffda38ea559b0612cf1 P 2cc5da2e62d84a2db3307053d9460b4f: Bootstrap starting.
I20260926 11:30:06.688937 4949 tablet_bootstrap.cc:492] T 068ab849591a4ffda38ea559b0612cf1 P 3e2eccf86bf34919aeb249bac4dca518: Bootstrap starting.
I20260926 11:30:06.689538 4948 tablet_bootstrap.cc:654] T 068ab849591a4ffda38ea559b0612cf1 P 4f1b6a1a2adc4f4dbfb8f1c9b03ab658: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:06.690743 4950 tablet_bootstrap.cc:654] T 068ab849591a4ffda38ea559b0612cf1 P 2cc5da2e62d84a2db3307053d9460b4f: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:06.690753 4949 tablet_bootstrap.cc:654] T 068ab849591a4ffda38ea559b0612cf1 P 3e2eccf86bf34919aeb249bac4dca518: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:06.690773 4948 log.cc:824] T 068ab849591a4ffda38ea559b0612cf1 P 4f1b6a1a2adc4f4dbfb8f1c9b03ab658: Log is configured to *not* fsync() on all Append() calls
I20260926 11:30:06.691900 4950 log.cc:824] T 068ab849591a4ffda38ea559b0612cf1 P 2cc5da2e62d84a2db3307053d9460b4f: Log is configured to *not* fsync() on all Append() calls
I20260926 11:30:06.691952 4949 log.cc:824] T 068ab849591a4ffda38ea559b0612cf1 P 3e2eccf86bf34919aeb249bac4dca518: Log is configured to *not* fsync() on all Append() calls
I20260926 11:30:06.692899 4948 tablet_bootstrap.cc:492] T 068ab849591a4ffda38ea559b0612cf1 P 4f1b6a1a2adc4f4dbfb8f1c9b03ab658: No bootstrap required, opened a new log
I20260926 11:30:06.693096 4948 ts_tablet_manager.cc:1397] T 068ab849591a4ffda38ea559b0612cf1 P 4f1b6a1a2adc4f4dbfb8f1c9b03ab658: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 11:30:06.693719 4950 tablet_bootstrap.cc:492] T 068ab849591a4ffda38ea559b0612cf1 P 2cc5da2e62d84a2db3307053d9460b4f: No bootstrap required, opened a new log
I20260926 11:30:06.693723 4949 tablet_bootstrap.cc:492] T 068ab849591a4ffda38ea559b0612cf1 P 3e2eccf86bf34919aeb249bac4dca518: No bootstrap required, opened a new log
I20260926 11:30:06.694033 4949 ts_tablet_manager.cc:1397] T 068ab849591a4ffda38ea559b0612cf1 P 3e2eccf86bf34919aeb249bac4dca518: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.004s
I20260926 11:30:06.693905 4950 ts_tablet_manager.cc:1397] T 068ab849591a4ffda38ea559b0612cf1 P 2cc5da2e62d84a2db3307053d9460b4f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260926 11:30:06.696453 4948 raft_consensus.cc:348] T 068ab849591a4ffda38ea559b0612cf1 P 4f1b6a1a2adc4f4dbfb8f1c9b03ab658 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2eccf86bf34919aeb249bac4dca518" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44787 } } peers { permanent_uuid: "2cc5da2e62d84a2db3307053d9460b4f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38207 } } peers { permanent_uuid: "4f1b6a1a2adc4f4dbfb8f1c9b03ab658" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39631 } }
I20260926 11:30:06.696694 4948 raft_consensus.cc:374] T 068ab849591a4ffda38ea559b0612cf1 P 4f1b6a1a2adc4f4dbfb8f1c9b03ab658 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:06.696774 4948 raft_consensus.cc:729] T 068ab849591a4ffda38ea559b0612cf1 P 4f1b6a1a2adc4f4dbfb8f1c9b03ab658 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f1b6a1a2adc4f4dbfb8f1c9b03ab658, State: Initialized, Role: FOLLOWER
I20260926 11:30:06.697218 4948 consensus_queue.cc:260] T 068ab849591a4ffda38ea559b0612cf1 P 4f1b6a1a2adc4f4dbfb8f1c9b03ab658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2eccf86bf34919aeb249bac4dca518" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44787 } } peers { permanent_uuid: "2cc5da2e62d84a2db3307053d9460b4f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38207 } } peers { permanent_uuid: "4f1b6a1a2adc4f4dbfb8f1c9b03ab658" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39631 } }
I20260926 11:30:06.697863 4798 heartbeater.cc:503] Master 127.0.98.62:44743 was elected leader, sending a full tablet report...
I20260926 11:30:06.697809 4949 raft_consensus.cc:348] T 068ab849591a4ffda38ea559b0612cf1 P 3e2eccf86bf34919aeb249bac4dca518 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2eccf86bf34919aeb249bac4dca518" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44787 } } peers { permanent_uuid: "2cc5da2e62d84a2db3307053d9460b4f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38207 } } peers { permanent_uuid: "4f1b6a1a2adc4f4dbfb8f1c9b03ab658" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39631 } }
I20260926 11:30:06.698068 4949 raft_consensus.cc:374] T 068ab849591a4ffda38ea559b0612cf1 P 3e2eccf86bf34919aeb249bac4dca518 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:06.698130 4949 raft_consensus.cc:729] T 068ab849591a4ffda38ea559b0612cf1 P 3e2eccf86bf34919aeb249bac4dca518 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e2eccf86bf34919aeb249bac4dca518, State: Initialized, Role: FOLLOWER
I20260926 11:30:06.698237 4948 ts_tablet_manager.cc:1428] T 068ab849591a4ffda38ea559b0612cf1 P 4f1b6a1a2adc4f4dbfb8f1c9b03ab658: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 11:30:06.698515 4949 consensus_queue.cc:260] T 068ab849591a4ffda38ea559b0612cf1 P 3e2eccf86bf34919aeb249bac4dca518 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2eccf86bf34919aeb249bac4dca518" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44787 } } peers { permanent_uuid: "2cc5da2e62d84a2db3307053d9460b4f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38207 } } peers { permanent_uuid: "4f1b6a1a2adc4f4dbfb8f1c9b03ab658" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39631 } }
I20260926 11:30:06.698535 4950 raft_consensus.cc:348] T 068ab849591a4ffda38ea559b0612cf1 P 2cc5da2e62d84a2db3307053d9460b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2eccf86bf34919aeb249bac4dca518" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44787 } } peers { permanent_uuid: "2cc5da2e62d84a2db3307053d9460b4f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38207 } } peers { permanent_uuid: "4f1b6a1a2adc4f4dbfb8f1c9b03ab658" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39631 } }
I20260926 11:30:06.698765 4950 raft_consensus.cc:374] T 068ab849591a4ffda38ea559b0612cf1 P 2cc5da2e62d84a2db3307053d9460b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:06.698858 4950 raft_consensus.cc:729] T 068ab849591a4ffda38ea559b0612cf1 P 2cc5da2e62d84a2db3307053d9460b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cc5da2e62d84a2db3307053d9460b4f, State: Initialized, Role: FOLLOWER
I20260926 11:30:06.699229 4931 heartbeater.cc:503] Master 127.0.98.62:44743 was elected leader, sending a full tablet report...
I20260926 11:30:06.699422 4949 ts_tablet_manager.cc:1428] T 068ab849591a4ffda38ea559b0612cf1 P 3e2eccf86bf34919aeb249bac4dca518: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260926 11:30:06.699353 4950 consensus_queue.cc:260] T 068ab849591a4ffda38ea559b0612cf1 P 2cc5da2e62d84a2db3307053d9460b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2eccf86bf34919aeb249bac4dca518" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44787 } } peers { permanent_uuid: "2cc5da2e62d84a2db3307053d9460b4f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38207 } } peers { permanent_uuid: "4f1b6a1a2adc4f4dbfb8f1c9b03ab658" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39631 } }
I20260926 11:30:06.699944 4665 heartbeater.cc:503] Master 127.0.98.62:44743 was elected leader, sending a full tablet report...
I20260926 11:30:06.700287 4950 ts_tablet_manager.cc:1428] T 068ab849591a4ffda38ea559b0612cf1 P 2cc5da2e62d84a2db3307053d9460b4f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 11:30:06.702765 4955 raft_consensus.cc:482] T 068ab849591a4ffda38ea559b0612cf1 P 3e2eccf86bf34919aeb249bac4dca518 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:30:06.702919 4955 raft_consensus.cc:504] T 068ab849591a4ffda38ea559b0612cf1 P 3e2eccf86bf34919aeb249bac4dca518 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2eccf86bf34919aeb249bac4dca518" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44787 } } peers { permanent_uuid: "2cc5da2e62d84a2db3307053d9460b4f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38207 } } peers { permanent_uuid: "4f1b6a1a2adc4f4dbfb8f1c9b03ab658" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39631 } }
I20260926 11:30:06.704010 4955 leader_election.cc:288] T 068ab849591a4ffda38ea559b0612cf1 P 3e2eccf86bf34919aeb249bac4dca518 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cc5da2e62d84a2db3307053d9460b4f (127.0.98.1:38207), 4f1b6a1a2adc4f4dbfb8f1c9b03ab658 (127.0.98.2:39631)
I20260926 11:30:06.709221 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:06.704248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47541 (local address 127.0.98.1:38207)
0926 11:30:06.704401 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:06.704406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:06.704557 (+ 151us) server_negotiation.cc:407] Connection header received
0926 11:30:06.704719 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:06.704722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:06.704769 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:06.704847 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:30:06.705294 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.706075 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.707828 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.708483 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.708581 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:06.708641 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:06.709036 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 11:30:06.709108 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 11:30:06.709228 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:06.704158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38207 (local address 127.0.98.3:47541)
0926 11:30:06.704490 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:06.704506 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:06.704603 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:06.704861 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:06.704864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:06.704901 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:30:06.705179 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:06.705186 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.706185 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.706189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:06.707700 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:06.707711 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.708607 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.708611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:06.708829 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:06.708932 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 11:30:06.709014 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 11:30:06.709096 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":310,"spinlock_wait_cycles":27776}
I20260926 11:30:06.709231 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:06.704313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46917 (local address 127.0.98.2:39631)
0926 11:30:06.704453 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:06.704457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:06.704643 (+ 186us) server_negotiation.cc:407] Connection header received
0926 11:30:06.704731 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:06.704733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:06.704775 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:06.704847 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:30:06.705294 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.706076 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.707834 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.708472 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.708581 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:06.708641 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:06.709066 (+ 425us) server_negotiation.cc:299] Negotiation successful
0926 11:30:06.709136 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 11:30:06.709882 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:06.704241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39631 (local address 127.0.98.3:46917)
0926 11:30:06.704565 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:06.704577 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:06.704655 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:06.704861 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:06.704864 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:06.704902 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:30:06.705182 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:06.705186 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.706185 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.706189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:06.707700 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:06.707711 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.708607 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.708611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:06.708832 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:06.708931 (+ 99us) client_negotiation.cc:769] Sending connection context
0926 11:30:06.709665 (+ 734us) client_negotiation.cc:240] Negotiation successful
0926 11:30:06.709739 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":287,"spinlock_wait_cycles":14080,"thread_start_us":144,"threads_started":1}
I20260926 11:30:06.710058 4619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "068ab849591a4ffda38ea559b0612cf1" candidate_uuid: "3e2eccf86bf34919aeb249bac4dca518" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cc5da2e62d84a2db3307053d9460b4f" is_pre_election: true
I20260926 11:30:06.710355 4619 raft_consensus.cc:2454] T 068ab849591a4ffda38ea559b0612cf1 P 2cc5da2e62d84a2db3307053d9460b4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e2eccf86bf34919aeb249bac4dca518 in term 0.
I20260926 11:30:06.710438 4752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "068ab849591a4ffda38ea559b0612cf1" candidate_uuid: "3e2eccf86bf34919aeb249bac4dca518" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f1b6a1a2adc4f4dbfb8f1c9b03ab658" is_pre_election: true
I20260926 11:30:06.710690 4752 raft_consensus.cc:2454] T 068ab849591a4ffda38ea559b0612cf1 P 4f1b6a1a2adc4f4dbfb8f1c9b03ab658 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e2eccf86bf34919aeb249bac4dca518 in term 0.
I20260926 11:30:06.710850 4819 leader_election.cc:302] T 068ab849591a4ffda38ea559b0612cf1 P 3e2eccf86bf34919aeb249bac4dca518 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cc5da2e62d84a2db3307053d9460b4f, 3e2eccf86bf34919aeb249bac4dca518; no voters:
I20260926 11:30:06.711087 4955 raft_consensus.cc:2781] T 068ab849591a4ffda38ea559b0612cf1 P 3e2eccf86bf34919aeb249bac4dca518 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:30:06.711164 4955 raft_consensus.cc:482] T 068ab849591a4ffda38ea559b0612cf1 P 3e2eccf86bf34919aeb249bac4dca518 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:30:06.711206 4955 raft_consensus.cc:3037] T 068ab849591a4ffda38ea559b0612cf1 P 3e2eccf86bf34919aeb249bac4dca518 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:06.712324 4955 raft_consensus.cc:504] T 068ab849591a4ffda38ea559b0612cf1 P 3e2eccf86bf34919aeb249bac4dca518 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2eccf86bf34919aeb249bac4dca518" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44787 } } peers { permanent_uuid: "2cc5da2e62d84a2db3307053d9460b4f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38207 } } peers { permanent_uuid: "4f1b6a1a2adc4f4dbfb8f1c9b03ab658" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39631 } }
I20260926 11:30:06.712743 4955 leader_election.cc:288] T 068ab849591a4ffda38ea559b0612cf1 P 3e2eccf86bf34919aeb249bac4dca518 [CANDIDATE]: Term 1 election: Requested vote from peers 2cc5da2e62d84a2db3307053d9460b4f (127.0.98.1:38207), 4f1b6a1a2adc4f4dbfb8f1c9b03ab658 (127.0.98.2:39631)
I20260926 11:30:06.713145 4752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "068ab849591a4ffda38ea559b0612cf1" candidate_uuid: "3e2eccf86bf34919aeb249bac4dca518" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f1b6a1a2adc4f4dbfb8f1c9b03ab658"
I20260926 11:30:06.713267 4752 raft_consensus.cc:3037] T 068ab849591a4ffda38ea559b0612cf1 P 4f1b6a1a2adc4f4dbfb8f1c9b03ab658 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:06.714818 4752 raft_consensus.cc:2454] T 068ab849591a4ffda38ea559b0612cf1 P 4f1b6a1a2adc4f4dbfb8f1c9b03ab658 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e2eccf86bf34919aeb249bac4dca518 in term 1.
I20260926 11:30:06.715112 4619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "068ab849591a4ffda38ea559b0612cf1" candidate_uuid: "3e2eccf86bf34919aeb249bac4dca518" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cc5da2e62d84a2db3307053d9460b4f"
I20260926 11:30:06.715200 4821 leader_election.cc:302] T 068ab849591a4ffda38ea559b0612cf1 P 3e2eccf86bf34919aeb249bac4dca518 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e2eccf86bf34919aeb249bac4dca518, 4f1b6a1a2adc4f4dbfb8f1c9b03ab658; no voters:
I20260926 11:30:06.715270 4619 raft_consensus.cc:3037] T 068ab849591a4ffda38ea559b0612cf1 P 2cc5da2e62d84a2db3307053d9460b4f [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:06.715448 4955 raft_consensus.cc:2781] T 068ab849591a4ffda38ea559b0612cf1 P 3e2eccf86bf34919aeb249bac4dca518 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:30:06.715703 4955 raft_consensus.cc:686] T 068ab849591a4ffda38ea559b0612cf1 P 3e2eccf86bf34919aeb249bac4dca518 [term 1 LEADER]: Becoming Leader. State: Replica: 3e2eccf86bf34919aeb249bac4dca518, State: Running, Role: LEADER
I20260926 11:30:06.716104 4955 consensus_queue.cc:237] T 068ab849591a4ffda38ea559b0612cf1 P 3e2eccf86bf34919aeb249bac4dca518 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2eccf86bf34919aeb249bac4dca518" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44787 } } peers { permanent_uuid: "2cc5da2e62d84a2db3307053d9460b4f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38207 } } peers { permanent_uuid: "4f1b6a1a2adc4f4dbfb8f1c9b03ab658" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39631 } }
I20260926 11:30:06.716729 4619 raft_consensus.cc:2454] T 068ab849591a4ffda38ea559b0612cf1 P 2cc5da2e62d84a2db3307053d9460b4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e2eccf86bf34919aeb249bac4dca518 in term 1.
I20260926 11:30:06.719560 4473 catalog_manager.cc:5495] T 068ab849591a4ffda38ea559b0612cf1 P 3e2eccf86bf34919aeb249bac4dca518 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e2eccf86bf34919aeb249bac4dca518 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3e2eccf86bf34919aeb249bac4dca518" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e2eccf86bf34919aeb249bac4dca518" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cc5da2e62d84a2db3307053d9460b4f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f1b6a1a2adc4f4dbfb8f1c9b03ab658" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39631 } health_report { overall_health: UNKNOWN } } }
I20260926 11:30:06.741446 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:06.737826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49380 (local address 127.0.98.3:44787)
0926 11:30:06.737957 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:06.737961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:06.737974 (+ 13us) server_negotiation.cc:407] Connection header received
0926 11:30:06.738088 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:06.738090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:06.738145 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:06.738288 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:30:06.738532 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.739030 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.739966 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:06.740176 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:06.740323 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:06.740641 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:30:06.741058 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:30:06.741194 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:06.741230 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 11:30:06.741294 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 11:30:06.747192 4752 raft_consensus.cc:1261] T 068ab849591a4ffda38ea559b0612cf1 P 4f1b6a1a2adc4f4dbfb8f1c9b03ab658 [term 1 FOLLOWER]: Refusing update from remote peer 3e2eccf86bf34919aeb249bac4dca518: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:30:06.747186 4619 raft_consensus.cc:1261] T 068ab849591a4ffda38ea559b0612cf1 P 2cc5da2e62d84a2db3307053d9460b4f [term 1 FOLLOWER]: Refusing update from remote peer 3e2eccf86bf34919aeb249bac4dca518: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:30:06.747957 4958 consensus_queue.cc:1035] T 068ab849591a4ffda38ea559b0612cf1 P 3e2eccf86bf34919aeb249bac4dca518 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f1b6a1a2adc4f4dbfb8f1c9b03ab658" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:30:06.748231 4955 consensus_queue.cc:1035] T 068ab849591a4ffda38ea559b0612cf1 P 3e2eccf86bf34919aeb249bac4dca518 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cc5da2e62d84a2db3307053d9460b4f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:30:06.755805 4962 mvcc.cc:205] Tried to move back new op lower bound from 7333569358829121536 to 7333569358712561664. Current Snapshot: MvccSnapshot[applied={T|T < 7333569358829121536}]
I20260926 11:30:06.763190 4473 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:30:06.763346 4473 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:30:06.765625 4473 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a100619781b344abbd835061961e80cd: Sending DeleteTablet for 3 replicas of tablet 068ab849591a4ffda38ea559b0612cf1
I20260926 11:30:06.766387 4599 tablet_service.cc:1514] Processing DeleteTablet for tablet 068ab849591a4ffda38ea559b0612cf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:30:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35680
I20260926 11:30:06.766424 4732 tablet_service.cc:1514] Processing DeleteTablet for tablet 068ab849591a4ffda38ea559b0612cf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:30:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32794
I20260926 11:30:06.766860 4865 tablet_service.cc:1514] Processing DeleteTablet for tablet 068ab849591a4ffda38ea559b0612cf1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:30:06 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49364
I20260926 11:30:06.767141 4972 tablet_replica.cc:330] T 068ab849591a4ffda38ea559b0612cf1 P 4f1b6a1a2adc4f4dbfb8f1c9b03ab658: stopping tablet replica
I20260926 11:30:06.767159 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 4537
I20260926 11:30:06.767460 4972 raft_consensus.cc:2229] T 068ab849591a4ffda38ea559b0612cf1 P 4f1b6a1a2adc4f4dbfb8f1c9b03ab658 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:30:06.767830 4972 raft_consensus.cc:2258] T 068ab849591a4ffda38ea559b0612cf1 P 4f1b6a1a2adc4f4dbfb8f1c9b03ab658 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:30:06.769305 4972 ts_tablet_manager.cc:1905] T 068ab849591a4ffda38ea559b0612cf1 P 4f1b6a1a2adc4f4dbfb8f1c9b03ab658: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:30:06.772610 4972 ts_tablet_manager.cc:1918] T 068ab849591a4ffda38ea559b0612cf1 P 4f1b6a1a2adc4f4dbfb8f1c9b03ab658: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:30:06.772711 4972 log.cc:1196] T 068ab849591a4ffda38ea559b0612cf1 P 4f1b6a1a2adc4f4dbfb8f1c9b03ab658: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790422180210734-392-0/minicluster-data/ts-1/wal/wals/068ab849591a4ffda38ea559b0612cf1
I20260926 11:30:06.773052 4972 ts_tablet_manager.cc:1939] T 068ab849591a4ffda38ea559b0612cf1 P 4f1b6a1a2adc4f4dbfb8f1c9b03ab658: Deleting consensus metadata
I20260926 11:30:06.774272 4462 catalog_manager.cc:4841] TS 4f1b6a1a2adc4f4dbfb8f1c9b03ab658 (127.0.98.2:39631): tablet 068ab849591a4ffda38ea559b0612cf1 (table test-table [id=e2e02e9938554e498d78aa8350921324]) successfully deleted
W20260926 11:30:06.776024 4460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38207 (error 108)
I20260926 11:30:06.776213 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 4670
I20260926 11:30:06.777909 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:06.777593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38207 (local address 127.0.0.1:35684)
0926 11:30:06.777737 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:06.777805 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260926 11:30:06.778136 4460 catalog_manager.cc:4584] TS 2cc5da2e62d84a2db3307053d9460b4f (127.0.98.1:38207): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 068ab849591a4ffda38ea559b0612cf1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38207: connect: Connection refused (error 111)
I20260926 11:30:06.782598 4973 tablet_replica.cc:330] T 068ab849591a4ffda38ea559b0612cf1 P 3e2eccf86bf34919aeb249bac4dca518: stopping tablet replica
I20260926 11:30:06.782618 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 4803
I20260926 11:30:06.782939 4973 raft_consensus.cc:2229] T 068ab849591a4ffda38ea559b0612cf1 P 3e2eccf86bf34919aeb249bac4dca518 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:30:06.787443 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 4444
2026-09-26T11:30:06Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2195 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T11:30:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:30:06Z Disabled control of system clock
I20260926 11:30:06.816481 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44251
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 11:30:06.924575 4978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:06.924849 4978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:06.924885 4978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:06.928903 4978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:30:06.928967 4978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:06.928983 4978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:06.929001 4978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:30:06.929051 4978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:30:06.933962 4978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44251
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:06.935106 4978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:06.936547 4978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:06.944620 4984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:06.944686 4986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:06.944639 4983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:06.944921 4978 server_base.cc:1034] running on GCE node
I20260926 11:30:06.945569 4978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:06.946532 4978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:06.947679 4978 hybrid_clock.cc:648] HybridClock initialized: now 1790422206947660 us; error 32 us; skew 500 ppm
I20260926 11:30:06.949506 4978 webserver.cc:460] Webserver started at http://127.0.98.62:34851/ using document root <none> and password file <none>
I20260926 11:30:06.950043 4978 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:06.950102 4978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:06.950299 4978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:06.952328 4978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "e11f46fa9ba84e749bc854739f7edbe5"
format_stamp: "Formatted at 2026-09-26 11:30:06 on dist-test-slave-mrck"
server_key: "0c51e51e70727d429f73e3ca69912b27"
server_key_iv: "eefb58f13d03a675fea1ae9b250cffb6"
server_key_version: "encryptionkey@0"
I20260926 11:30:06.952847 4978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "e11f46fa9ba84e749bc854739f7edbe5"
format_stamp: "Formatted at 2026-09-26 11:30:06 on dist-test-slave-mrck"
server_key: "0c51e51e70727d429f73e3ca69912b27"
server_key_iv: "eefb58f13d03a675fea1ae9b250cffb6"
server_key_version: "encryptionkey@0"
I20260926 11:30:06.956153 4978 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:30:06.958407 4992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:06.959522 4978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:30:06.959637 4978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "e11f46fa9ba84e749bc854739f7edbe5"
format_stamp: "Formatted at 2026-09-26 11:30:06 on dist-test-slave-mrck"
server_key: "0c51e51e70727d429f73e3ca69912b27"
server_key_iv: "eefb58f13d03a675fea1ae9b250cffb6"
server_key_version: "encryptionkey@0"
I20260926 11:30:06.959782 4978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:06.964097 4978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:06.964684 4978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:06.964866 4978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:06.971310 4978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44251
I20260926 11:30:06.971315 5044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44251 every 8 connection(s)
I20260926 11:30:06.972282 4978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:30:06.975291 5045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:06.980502 5045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e11f46fa9ba84e749bc854739f7edbe5: Bootstrap starting.
I20260926 11:30:06.981899 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 4978
I20260926 11:30:06.982019 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:30:06.982306 392 external_mini_cluster.cc:1428] Setting key 267bcf345a585768b559c9e043bb010d
I20260926 11:30:06.983229 5045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e11f46fa9ba84e749bc854739f7edbe5: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:06.984037 5045 log.cc:824] T 00000000000000000000000000000000 P e11f46fa9ba84e749bc854739f7edbe5: Log is configured to *not* fsync() on all Append() calls
I20260926 11:30:06.985891 5045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e11f46fa9ba84e749bc854739f7edbe5: No bootstrap required, opened a new log
I20260926 11:30:06.987277 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:06.983400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50784 (local address 127.0.98.62:44251)
0926 11:30:06.984064 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:06.984079 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:06.984111 (+ 32us) server_negotiation.cc:407] Connection header received
0926 11:30:06.984750 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:06.984766 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:06.985031 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:06.985364 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:06.986086 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:06.986091 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:06.986103 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:30:06.986112 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:06.986162 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:30:06.986177 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:06.986288 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:06.986387 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 11:30:06.986613 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":117,"threads_started":1}
I20260926 11:30:06.989462 5045 raft_consensus.cc:348] T 00000000000000000000000000000000 P e11f46fa9ba84e749bc854739f7edbe5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11f46fa9ba84e749bc854739f7edbe5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44251 } }
I20260926 11:30:06.989707 5045 raft_consensus.cc:374] T 00000000000000000000000000000000 P e11f46fa9ba84e749bc854739f7edbe5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:06.989780 5045 raft_consensus.cc:729] T 00000000000000000000000000000000 P e11f46fa9ba84e749bc854739f7edbe5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e11f46fa9ba84e749bc854739f7edbe5, State: Initialized, Role: FOLLOWER
I20260926 11:30:06.990222 5045 consensus_queue.cc:260] T 00000000000000000000000000000000 P e11f46fa9ba84e749bc854739f7edbe5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11f46fa9ba84e749bc854739f7edbe5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44251 } }
I20260926 11:30:06.990365 5045 raft_consensus.cc:388] T 00000000000000000000000000000000 P e11f46fa9ba84e749bc854739f7edbe5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:30:06.990447 5045 raft_consensus.cc:482] T 00000000000000000000000000000000 P e11f46fa9ba84e749bc854739f7edbe5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:30:06.990545 5045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e11f46fa9ba84e749bc854739f7edbe5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:06.991359 5045 raft_consensus.cc:504] T 00000000000000000000000000000000 P e11f46fa9ba84e749bc854739f7edbe5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11f46fa9ba84e749bc854739f7edbe5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44251 } }
I20260926 11:30:06.991675 5045 leader_election.cc:302] T 00000000000000000000000000000000 P e11f46fa9ba84e749bc854739f7edbe5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e11f46fa9ba84e749bc854739f7edbe5; no voters:
I20260926 11:30:06.992045 5045 leader_election.cc:288] T 00000000000000000000000000000000 P e11f46fa9ba84e749bc854739f7edbe5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:30:06.992108 5050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e11f46fa9ba84e749bc854739f7edbe5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:30:06.992336 5050 raft_consensus.cc:686] T 00000000000000000000000000000000 P e11f46fa9ba84e749bc854739f7edbe5 [term 1 LEADER]: Becoming Leader. State: Replica: e11f46fa9ba84e749bc854739f7edbe5, State: Running, Role: LEADER
I20260926 11:30:06.992632 5050 consensus_queue.cc:237] T 00000000000000000000000000000000 P e11f46fa9ba84e749bc854739f7edbe5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11f46fa9ba84e749bc854739f7edbe5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44251 } }
I20260926 11:30:06.993345 5045 sys_catalog.cc:564] T 00000000000000000000000000000000 P e11f46fa9ba84e749bc854739f7edbe5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:30:06.994390 5051 sys_catalog.cc:455] T 00000000000000000000000000000000 P e11f46fa9ba84e749bc854739f7edbe5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e11f46fa9ba84e749bc854739f7edbe5. Latest consensus state: current_term: 1 leader_uuid: "e11f46fa9ba84e749bc854739f7edbe5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11f46fa9ba84e749bc854739f7edbe5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44251 } } }
I20260926 11:30:06.994627 5051 sys_catalog.cc:458] T 00000000000000000000000000000000 P e11f46fa9ba84e749bc854739f7edbe5 [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:06.994431 5052 sys_catalog.cc:455] T 00000000000000000000000000000000 P e11f46fa9ba84e749bc854739f7edbe5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e11f46fa9ba84e749bc854739f7edbe5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11f46fa9ba84e749bc854739f7edbe5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44251 } } }
I20260926 11:30:06.994752 5052 sys_catalog.cc:458] T 00000000000000000000000000000000 P e11f46fa9ba84e749bc854739f7edbe5 [sys.catalog]: This master's current role is: LEADER
W20260926 11:30:06.998155 5065 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e11f46fa9ba84e749bc854739f7edbe5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:30:06.998240 5065 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:30:06.998306 5060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:30:06.998955 5060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:30:07.003787 5060 catalog_manager.cc:1347] Generated new cluster ID: 92487acceee24f2789462134c4c4831d
I20260926 11:30:07.003851 5060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:30:07.010238 5060 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:30:07.010560 5060 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:30:07.017030 5060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e11f46fa9ba84e749bc854739f7edbe5: Generated new TSK 0
I20260926 11:30:07.017673 5060 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:30:07.026592 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44251
--builtin_ntp_servers=127.0.98.20:36371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 11:30:07.137780 5069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:07.138041 5069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:07.138080 5069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:07.142119 5069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:07.142162 5069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:07.142246 5069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:30:07.146828 5069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:07.147851 5069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:07.149175 5069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:07.158320 5075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:07.158334 5077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:07.158341 5074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:07.158736 5069 server_base.cc:1034] running on GCE node
I20260926 11:30:07.159112 5069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:07.159684 5069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:07.160912 5069 hybrid_clock.cc:648] HybridClock initialized: now 1790422207160895 us; error 26 us; skew 500 ppm
I20260926 11:30:07.162817 5069 webserver.cc:460] Webserver started at http://127.0.98.1:38405/ using document root <none> and password file <none>
I20260926 11:30:07.163398 5069 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:07.163506 5069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:07.163762 5069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:07.165760 5069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-0/data/instance:
uuid: "d86646389005421ca729b7c72133d8e7"
format_stamp: "Formatted at 2026-09-26 11:30:07 on dist-test-slave-mrck"
server_key: "28e503c004dee9ac24847d844c1158f1"
server_key_iv: "fe46fa9250aafcda28469dd53b6b63d6"
server_key_version: "encryptionkey@0"
I20260926 11:30:07.166329 5069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d86646389005421ca729b7c72133d8e7"
format_stamp: "Formatted at 2026-09-26 11:30:07 on dist-test-slave-mrck"
server_key: "28e503c004dee9ac24847d844c1158f1"
server_key_iv: "fe46fa9250aafcda28469dd53b6b63d6"
server_key_version: "encryptionkey@0"
I20260926 11:30:07.169898 5069 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260926 11:30:07.172320 5083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:07.173517 5069 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:30:07.173656 5069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "d86646389005421ca729b7c72133d8e7"
format_stamp: "Formatted at 2026-09-26 11:30:07 on dist-test-slave-mrck"
server_key: "28e503c004dee9ac24847d844c1158f1"
server_key_iv: "fe46fa9250aafcda28469dd53b6b63d6"
server_key_version: "encryptionkey@0"
I20260926 11:30:07.173772 5069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:07.178408 5069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:07.179013 5069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:07.179199 5069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:07.179765 5069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:07.180671 5069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:07.180740 5069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:07.180814 5069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:07.180864 5069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:07.189132 5069 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35997
I20260926 11:30:07.189196 5196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35997 every 8 connection(s)
I20260926 11:30:07.190042 5069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:30:07.192569 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 5069
I20260926 11:30:07.192706 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:30:07.193038 392 external_mini_cluster.cc:1428] Setting key 02cf29ea2ef4c3860eae57ae663b72db
I20260926 11:30:07.195318 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44251
--builtin_ntp_servers=127.0.98.20:36371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 11:30:07.196725 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:07.191837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50377 (local address 127.0.98.62:44251)
0926 11:30:07.191967 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:07.191971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:07.193273 (+ 1302us) server_negotiation.cc:407] Connection header received
0926 11:30:07.194908 (+ 1635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:07.194912 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:07.194948 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:07.195020 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:07.196252 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:07.196256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:07.196259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:30:07.196270 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:07.196291 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:30:07.196302 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:07.196418 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:07.196560 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 11:30:07.196600 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 11:30:07.197592 5199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:07.192278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44251 (local address 127.0.98.1:50377)
0926 11:30:07.193113 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:07.193166 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:07.194691 (+ 1525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:07.195199 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:07.195205 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:07.195294 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:07.196012 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:30:07.196014 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:07.196053 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:30:07.196118 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:07.196405 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:07.196414 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:07.196419 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 11:30:07.196593 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 11:30:07.196824 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":137,"threads_started":1}
I20260926 11:30:07.198763 5197 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44251
I20260926 11:30:07.199010 5197 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:07.199528 5197 heartbeater.cc:511] Master 127.0.98.62:44251 requested a full tablet report, sending...
I20260926 11:30:07.201139 5009 ts_manager.cc:194] Registered new tserver with Master: d86646389005421ca729b7c72133d8e7 (127.0.98.1:35997)
W20260926 11:30:07.305284 5200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:07.305575 5200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:07.305646 5200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:07.309623 5200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:07.309705 5200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:07.309798 5200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:30:07.314373 5200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:07.315470 5200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:07.316908 5200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:07.326439 5208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:07.326432 5205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:07.326407 5206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:07.326860 5200 server_base.cc:1034] running on GCE node
I20260926 11:30:07.327261 5200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:07.327947 5200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:07.329145 5200 hybrid_clock.cc:648] HybridClock initialized: now 1790422207329128 us; error 31 us; skew 500 ppm
I20260926 11:30:07.331048 5200 webserver.cc:460] Webserver started at http://127.0.98.2:40439/ using document root <none> and password file <none>
I20260926 11:30:07.331619 5200 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:07.331709 5200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:07.331972 5200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:07.334002 5200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-1/data/instance:
uuid: "04de65faee334f4ab840af5b889414f8"
format_stamp: "Formatted at 2026-09-26 11:30:07 on dist-test-slave-mrck"
server_key: "21dd7961daa724095135fa01aa3e303d"
server_key_iv: "6eb72657ee2677e3ddeaa890cab48bfe"
server_key_version: "encryptionkey@0"
I20260926 11:30:07.334574 5200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "04de65faee334f4ab840af5b889414f8"
format_stamp: "Formatted at 2026-09-26 11:30:07 on dist-test-slave-mrck"
server_key: "21dd7961daa724095135fa01aa3e303d"
server_key_iv: "6eb72657ee2677e3ddeaa890cab48bfe"
server_key_version: "encryptionkey@0"
I20260926 11:30:07.338361 5200 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 11:30:07.340875 5214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:07.342170 5200 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:30:07.342303 5200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "04de65faee334f4ab840af5b889414f8"
format_stamp: "Formatted at 2026-09-26 11:30:07 on dist-test-slave-mrck"
server_key: "21dd7961daa724095135fa01aa3e303d"
server_key_iv: "6eb72657ee2677e3ddeaa890cab48bfe"
server_key_version: "encryptionkey@0"
I20260926 11:30:07.342417 5200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:07.346918 5200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:07.347527 5200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:07.347707 5200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:07.348345 5200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:07.349251 5200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:07.349319 5200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:07.349388 5200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:07.349439 5200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:07.358157 5200 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41673
I20260926 11:30:07.358175 5327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41673 every 8 connection(s)
I20260926 11:30:07.359053 5200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:30:07.361526 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 5200
I20260926 11:30:07.361654 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:30:07.361964 392 external_mini_cluster.cc:1428] Setting key 0bf7534bf08d0e237b1fd02b80141a17
I20260926 11:30:07.364166 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44251
--builtin_ntp_servers=127.0.98.20:36371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 11:30:07.365738 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:07.360809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45005 (local address 127.0.98.62:44251)
0926 11:30:07.360971 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:07.360976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:07.362256 (+ 1280us) server_negotiation.cc:407] Connection header received
0926 11:30:07.363877 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:07.363880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:07.363912 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:07.363972 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:07.365177 (+ 1205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:07.365180 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:07.365182 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:30:07.365193 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:07.365212 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:30:07.365220 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:07.365311 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:07.365511 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 11:30:07.365564 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 11:30:07.366542 5330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:07.361278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44251 (local address 127.0.98.2:45005)
0926 11:30:07.362079 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:07.362135 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:07.363638 (+ 1503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:07.364157 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:07.364167 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:07.364286 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:07.364860 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:30:07.364864 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:07.364920 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:30:07.365021 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:07.365308 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:07.365318 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:07.365327 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 11:30:07.365513 (+ 186us) client_negotiation.cc:240] Negotiation successful
0926 11:30:07.365757 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":106,"threads_started":1}
I20260926 11:30:07.367699 5328 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44251
I20260926 11:30:07.367997 5328 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:07.368407 5328 heartbeater.cc:511] Master 127.0.98.62:44251 requested a full tablet report, sending...
I20260926 11:30:07.369462 5009 ts_manager.cc:194] Registered new tserver with Master: 04de65faee334f4ab840af5b889414f8 (127.0.98.2:41673)
W20260926 11:30:07.474646 5331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:07.474912 5331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:07.474951 5331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:07.479089 5331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:07.479146 5331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:07.479202 5331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:30:07.483923 5331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:07.484944 5331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:07.486269 5331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:07.495893 5336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:07.496022 5337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:07.496050 5339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:07.496037 5331 server_base.cc:1034] running on GCE node
I20260926 11:30:07.496516 5331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:07.497143 5331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:07.498302 5331 hybrid_clock.cc:648] HybridClock initialized: now 1790422207498278 us; error 35 us; skew 500 ppm
I20260926 11:30:07.500155 5331 webserver.cc:460] Webserver started at http://127.0.98.3:35709/ using document root <none> and password file <none>
I20260926 11:30:07.500725 5331 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:07.500784 5331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:07.500988 5331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:07.502938 5331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-2/data/instance:
uuid: "9310fb3e9a984873a5c72d2273da0545"
format_stamp: "Formatted at 2026-09-26 11:30:07 on dist-test-slave-mrck"
server_key: "1e6146a595554e79e03125fedc80d42c"
server_key_iv: "0c8acbe87cc2deeea5e41f63c3fc6efb"
server_key_version: "encryptionkey@0"
I20260926 11:30:07.503479 5331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9310fb3e9a984873a5c72d2273da0545"
format_stamp: "Formatted at 2026-09-26 11:30:07 on dist-test-slave-mrck"
server_key: "1e6146a595554e79e03125fedc80d42c"
server_key_iv: "0c8acbe87cc2deeea5e41f63c3fc6efb"
server_key_version: "encryptionkey@0"
I20260926 11:30:07.507005 5331 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 11:30:07.509438 5345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:07.510708 5331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 11:30:07.510840 5331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "9310fb3e9a984873a5c72d2273da0545"
format_stamp: "Formatted at 2026-09-26 11:30:07 on dist-test-slave-mrck"
server_key: "1e6146a595554e79e03125fedc80d42c"
server_key_iv: "0c8acbe87cc2deeea5e41f63c3fc6efb"
server_key_version: "encryptionkey@0"
I20260926 11:30:07.510950 5331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:07.515462 5331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:07.516109 5331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:07.516292 5331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:07.516839 5331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:07.517735 5331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:07.517805 5331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:07.517874 5331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:07.517920 5331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:07.526705 5331 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43449
I20260926 11:30:07.526726 5458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43449 every 8 connection(s)
I20260926 11:30:07.527580 5331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:30:07.530627 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 5331
I20260926 11:30:07.530750 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:30:07.531042 392 external_mini_cluster.cc:1428] Setting key 344b6c8fbf7f6453ca1b0fd4f6aafe06
I20260926 11:30:07.534051 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:07.529394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35445 (local address 127.0.98.62:44251)
0926 11:30:07.529536 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:07.529540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:07.530634 (+ 1094us) server_negotiation.cc:407] Connection header received
0926 11:30:07.532311 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:07.532315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:07.532357 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:07.532462 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:07.533577 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:07.533580 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:07.533583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:30:07.533594 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:07.533617 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:30:07.533627 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:07.533727 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:07.533873 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 11:30:07.533916 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 11:30:07.534837 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:07.529789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44251 (local address 127.0.98.3:35445)
0926 11:30:07.530457 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:07.530515 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:07.532110 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:07.532635 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:07.532641 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:07.532730 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:07.533317 (+ 587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:30:07.533319 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:07.533358 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:30:07.533428 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:07.533717 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:07.533725 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:07.533731 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 11:30:07.533904 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 11:30:07.534135 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":106,"threads_started":1}
I20260926 11:30:07.535936 5459 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44251
I20260926 11:30:07.536154 5459 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:07.536540 5459 heartbeater.cc:511] Master 127.0.98.62:44251 requested a full tablet report, sending...
I20260926 11:30:07.537505 5009 ts_manager.cc:194] Registered new tserver with Master: 9310fb3e9a984873a5c72d2273da0545 (127.0.98.3:43449)
I20260926 11:30:07.544867 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:30:07.548382 5048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:07.546820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50790 (local address 127.0.98.62:44251)
0926 11:30:07.546969 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:07.546974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:07.547153 (+ 179us) server_negotiation.cc:407] Connection header received
0926 11:30:07.547247 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:07.547249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:07.547281 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:07.547345 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:07.548031 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:07.548034 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:07.548036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:30:07.548045 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:07.548063 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:30:07.548070 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:07.548142 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:07.548185 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 11:30:07.548216 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 11:30:07.550760 5009 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:30:07.553845 5009 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:30:07.565434 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:07.561863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41673 (local address 127.0.0.1:53188)
0926 11:30:07.562721 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:07.562770 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:07.562945 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:07.563515 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:07.563522 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:07.563598 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:07.564206 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:30:07.564210 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:07.564232 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:30:07.564305 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:07.565087 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:07.565089 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:07.565092 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 11:30:07.565202 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 11:30:07.565270 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":75,"threads_started":1}
I20260926 11:30:07.565482 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:07.561989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53188 (local address 127.0.98.2:41673)
0926 11:30:07.562311 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:07.562326 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:07.562866 (+ 540us) server_negotiation.cc:407] Connection header received
0926 11:30:07.563081 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:07.563089 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:07.563296 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:07.563439 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:07.564418 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:07.564426 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:07.564440 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:30:07.564855 (+ 415us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:07.564921 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:30:07.564967 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:07.565069 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:07.565228 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 11:30:07.565303 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":141,"threads_started":1}
I20260926 11:30:07.565963 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:07.562541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43449 (local address 127.0.0.1:41728)
0926 11:30:07.563343 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:07.563354 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:07.563452 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:07.563951 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:07.563954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:07.563967 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:07.564680 (+ 713us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:30:07.564681 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:07.564688 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:30:07.564707 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:07.565010 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:07.565018 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:07.565023 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 11:30:07.565792 (+ 769us) client_negotiation.cc:240] Negotiation successful
0926 11:30:07.565836 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":82,"threads_started":1}
I20260926 11:30:07.566079 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:07.562914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41728 (local address 127.0.98.3:43449)
0926 11:30:07.563271 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:07.563282 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:07.563552 (+ 270us) server_negotiation.cc:407] Connection header received
0926 11:30:07.563600 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:07.563605 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:07.563826 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:07.563964 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:07.564816 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:07.564821 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:07.564830 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:30:07.564840 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:07.564884 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:30:07.564922 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:07.565035 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:07.565837 (+ 802us) server_negotiation.cc:299] Negotiation successful
0926 11:30:07.565906 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":154,"threads_started":1}
I20260926 11:30:07.566221 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:07.561912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35997 (local address 127.0.0.1:46796)
0926 11:30:07.563162 (+ 1250us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:07.563173 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:07.563237 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:07.564348 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:07.564351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:07.564368 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:07.564872 (+ 504us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:30:07.564873 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:07.564882 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:30:07.564925 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:07.565333 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:07.565336 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:07.565339 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 11:30:07.565408 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 11:30:07.565437 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1193,"thread_start_us":86,"threads_started":1}
I20260926 11:30:07.566553 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:07.562511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46796 (local address 127.0.98.1:35997)
0926 11:30:07.562891 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:07.562903 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:07.563325 (+ 422us) server_negotiation.cc:407] Connection header received
0926 11:30:07.563385 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:07.563392 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:07.563563 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:07.563684 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:07.565077 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:07.565086 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:07.565096 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:30:07.565108 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:07.565157 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:30:07.565203 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:07.565294 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:07.565896 (+ 602us) server_negotiation.cc:299] Negotiation successful
0926 11:30:07.565950 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":96,"threads_started":1}
I20260926 11:30:07.568738 5393 tablet_service.cc:1467] Processing CreateTablet for tablet 8827a39dde394a1fb58bc76933cd46ff (DEFAULT_TABLE table=test-table [id=193320f7b5cc472da252ab56f8a8dd06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:30:07.568706 5262 tablet_service.cc:1467] Processing CreateTablet for tablet 8827a39dde394a1fb58bc76933cd46ff (DEFAULT_TABLE table=test-table [id=193320f7b5cc472da252ab56f8a8dd06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:30:07.569033 5131 tablet_service.cc:1467] Processing CreateTablet for tablet 8827a39dde394a1fb58bc76933cd46ff (DEFAULT_TABLE table=test-table [id=193320f7b5cc472da252ab56f8a8dd06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:30:07.569692 5393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8827a39dde394a1fb58bc76933cd46ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:07.569715 5262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8827a39dde394a1fb58bc76933cd46ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:07.569846 5131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8827a39dde394a1fb58bc76933cd46ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:07.575249 5476 tablet_bootstrap.cc:492] T 8827a39dde394a1fb58bc76933cd46ff P 9310fb3e9a984873a5c72d2273da0545: Bootstrap starting.
I20260926 11:30:07.577253 5478 tablet_bootstrap.cc:492] T 8827a39dde394a1fb58bc76933cd46ff P d86646389005421ca729b7c72133d8e7: Bootstrap starting.
I20260926 11:30:07.577847 5476 tablet_bootstrap.cc:654] T 8827a39dde394a1fb58bc76933cd46ff P 9310fb3e9a984873a5c72d2273da0545: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:07.578660 5476 log.cc:824] T 8827a39dde394a1fb58bc76933cd46ff P 9310fb3e9a984873a5c72d2273da0545: Log is configured to *not* fsync() on all Append() calls
I20260926 11:30:07.579193 5478 tablet_bootstrap.cc:654] T 8827a39dde394a1fb58bc76933cd46ff P d86646389005421ca729b7c72133d8e7: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:07.579279 5477 tablet_bootstrap.cc:492] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8: Bootstrap starting.
I20260926 11:30:07.580019 5478 log.cc:824] T 8827a39dde394a1fb58bc76933cd46ff P d86646389005421ca729b7c72133d8e7: Log is configured to *not* fsync() on all Append() calls
I20260926 11:30:07.580889 5476 tablet_bootstrap.cc:492] T 8827a39dde394a1fb58bc76933cd46ff P 9310fb3e9a984873a5c72d2273da0545: No bootstrap required, opened a new log
I20260926 11:30:07.581084 5476 ts_tablet_manager.cc:1397] T 8827a39dde394a1fb58bc76933cd46ff P 9310fb3e9a984873a5c72d2273da0545: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 11:30:07.581313 5477 tablet_bootstrap.cc:654] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:07.581920 5478 tablet_bootstrap.cc:492] T 8827a39dde394a1fb58bc76933cd46ff P d86646389005421ca729b7c72133d8e7: No bootstrap required, opened a new log
I20260926 11:30:07.582103 5478 ts_tablet_manager.cc:1397] T 8827a39dde394a1fb58bc76933cd46ff P d86646389005421ca729b7c72133d8e7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 11:30:07.582079 5477 log.cc:824] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8: Log is configured to *not* fsync() on all Append() calls
I20260926 11:30:07.584064 5477 tablet_bootstrap.cc:492] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8: No bootstrap required, opened a new log
I20260926 11:30:07.584245 5477 ts_tablet_manager.cc:1397] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 11:30:07.585157 5476 raft_consensus.cc:348] T 8827a39dde394a1fb58bc76933cd46ff P 9310fb3e9a984873a5c72d2273da0545 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04de65faee334f4ab840af5b889414f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41673 } } peers { permanent_uuid: "d86646389005421ca729b7c72133d8e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35997 } } peers { permanent_uuid: "9310fb3e9a984873a5c72d2273da0545" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43449 } }
I20260926 11:30:07.585433 5476 raft_consensus.cc:374] T 8827a39dde394a1fb58bc76933cd46ff P 9310fb3e9a984873a5c72d2273da0545 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:07.585486 5476 raft_consensus.cc:729] T 8827a39dde394a1fb58bc76933cd46ff P 9310fb3e9a984873a5c72d2273da0545 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9310fb3e9a984873a5c72d2273da0545, State: Initialized, Role: FOLLOWER
I20260926 11:30:07.585934 5476 consensus_queue.cc:260] T 8827a39dde394a1fb58bc76933cd46ff P 9310fb3e9a984873a5c72d2273da0545 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04de65faee334f4ab840af5b889414f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41673 } } peers { permanent_uuid: "d86646389005421ca729b7c72133d8e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35997 } } peers { permanent_uuid: "9310fb3e9a984873a5c72d2273da0545" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43449 } }
I20260926 11:30:07.586565 5459 heartbeater.cc:503] Master 127.0.98.62:44251 was elected leader, sending a full tablet report...
I20260926 11:30:07.586670 5478 raft_consensus.cc:348] T 8827a39dde394a1fb58bc76933cd46ff P d86646389005421ca729b7c72133d8e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04de65faee334f4ab840af5b889414f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41673 } } peers { permanent_uuid: "d86646389005421ca729b7c72133d8e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35997 } } peers { permanent_uuid: "9310fb3e9a984873a5c72d2273da0545" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43449 } }
I20260926 11:30:07.586944 5478 raft_consensus.cc:374] T 8827a39dde394a1fb58bc76933cd46ff P d86646389005421ca729b7c72133d8e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:07.587023 5478 raft_consensus.cc:729] T 8827a39dde394a1fb58bc76933cd46ff P d86646389005421ca729b7c72133d8e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d86646389005421ca729b7c72133d8e7, State: Initialized, Role: FOLLOWER
I20260926 11:30:07.587517 5478 consensus_queue.cc:260] T 8827a39dde394a1fb58bc76933cd46ff P d86646389005421ca729b7c72133d8e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04de65faee334f4ab840af5b889414f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41673 } } peers { permanent_uuid: "d86646389005421ca729b7c72133d8e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35997 } } peers { permanent_uuid: "9310fb3e9a984873a5c72d2273da0545" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43449 } }
I20260926 11:30:07.588025 5476 ts_tablet_manager.cc:1428] T 8827a39dde394a1fb58bc76933cd46ff P 9310fb3e9a984873a5c72d2273da0545: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 11:30:07.588490 5478 ts_tablet_manager.cc:1428] T 8827a39dde394a1fb58bc76933cd46ff P d86646389005421ca729b7c72133d8e7: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260926 11:30:07.588842 5197 heartbeater.cc:503] Master 127.0.98.62:44251 was elected leader, sending a full tablet report...
I20260926 11:30:07.588729 5477 raft_consensus.cc:348] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04de65faee334f4ab840af5b889414f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41673 } } peers { permanent_uuid: "d86646389005421ca729b7c72133d8e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35997 } } peers { permanent_uuid: "9310fb3e9a984873a5c72d2273da0545" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43449 } }
I20260926 11:30:07.588923 5477 raft_consensus.cc:374] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:07.588968 5477 raft_consensus.cc:729] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04de65faee334f4ab840af5b889414f8, State: Initialized, Role: FOLLOWER
I20260926 11:30:07.589309 5477 consensus_queue.cc:260] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04de65faee334f4ab840af5b889414f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41673 } } peers { permanent_uuid: "d86646389005421ca729b7c72133d8e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35997 } } peers { permanent_uuid: "9310fb3e9a984873a5c72d2273da0545" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43449 } }
I20260926 11:30:07.590036 5328 heartbeater.cc:503] Master 127.0.98.62:44251 was elected leader, sending a full tablet report...
I20260926 11:30:07.590260 5477 ts_tablet_manager.cc:1428] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260926 11:30:07.609949 5329 tablet.cc:2369] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:30:07.611629 5484 raft_consensus.cc:482] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:30:07.611843 5484 raft_consensus.cc:504] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04de65faee334f4ab840af5b889414f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41673 } } peers { permanent_uuid: "d86646389005421ca729b7c72133d8e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35997 } } peers { permanent_uuid: "9310fb3e9a984873a5c72d2273da0545" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43449 } }
I20260926 11:30:07.612882 5484 leader_election.cc:288] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d86646389005421ca729b7c72133d8e7 (127.0.98.1:35997), 9310fb3e9a984873a5c72d2273da0545 (127.0.98.3:43449)
I20260926 11:30:07.614765 5330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:07.613136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35997 (local address 127.0.98.2:51225)
0926 11:30:07.613251 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:07.613268 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:07.613350 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:07.613683 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:07.613689 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:07.613704 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:07.614236 (+ 532us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:30:07.614239 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:07.614252 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:30:07.614287 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:07.614533 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:07.614535 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:07.614537 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 11:30:07.614602 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 11:30:07.614633 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 11:30:07.615015 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:07.613353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51225 (local address 127.0.98.1:35997)
0926 11:30:07.613472 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:07.613478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:07.613496 (+ 18us) server_negotiation.cc:407] Connection header received
0926 11:30:07.613534 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:07.613536 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:07.613564 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:07.613636 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:07.614417 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:07.614422 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:07.614426 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:30:07.614443 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:07.614461 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:30:07.614469 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:07.614542 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:07.614627 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 11:30:07.614669 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 11:30:07.615278 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:07.613136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43449 (local address 127.0.98.2:50115)
0926 11:30:07.613593 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:07.613604 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:07.613748 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:07.614017 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:07.614020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:07.614034 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:07.614626 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:30:07.614628 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:07.614639 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:30:07.614665 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:07.615014 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:07.615017 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:07.615020 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 11:30:07.615091 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 11:30:07.615140 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":128,"thread_start_us":127,"threads_started":1}
I20260926 11:30:07.615411 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:07.613250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50115 (local address 127.0.98.3:43449)
0926 11:30:07.613429 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:07.613434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:07.613713 (+ 279us) server_negotiation.cc:407] Connection header received
0926 11:30:07.613854 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:07.613858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:07.613899 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:07.613995 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:07.614815 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:07.614819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:07.614822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:30:07.614836 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:07.614857 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:30:07.614869 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:07.614976 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:07.615226 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 11:30:07.615279 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260926 11:30:07.615628 5151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8827a39dde394a1fb58bc76933cd46ff" candidate_uuid: "04de65faee334f4ab840af5b889414f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d86646389005421ca729b7c72133d8e7" is_pre_election: true
I20260926 11:30:07.615971 5151 raft_consensus.cc:2454] T 8827a39dde394a1fb58bc76933cd46ff P d86646389005421ca729b7c72133d8e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04de65faee334f4ab840af5b889414f8 in term 0.
I20260926 11:30:07.615971 5413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8827a39dde394a1fb58bc76933cd46ff" candidate_uuid: "04de65faee334f4ab840af5b889414f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9310fb3e9a984873a5c72d2273da0545" is_pre_election: true
I20260926 11:30:07.616245 5413 raft_consensus.cc:2454] T 8827a39dde394a1fb58bc76933cd46ff P 9310fb3e9a984873a5c72d2273da0545 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04de65faee334f4ab840af5b889414f8 in term 0.
I20260926 11:30:07.616463 5215 leader_election.cc:302] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04de65faee334f4ab840af5b889414f8, d86646389005421ca729b7c72133d8e7; no voters:
I20260926 11:30:07.616779 5484 raft_consensus.cc:2781] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:30:07.616880 5484 raft_consensus.cc:482] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:30:07.616933 5484 raft_consensus.cc:3037] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:07.617949 5484 raft_consensus.cc:504] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04de65faee334f4ab840af5b889414f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41673 } } peers { permanent_uuid: "d86646389005421ca729b7c72133d8e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35997 } } peers { permanent_uuid: "9310fb3e9a984873a5c72d2273da0545" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43449 } }
I20260926 11:30:07.618309 5484 leader_election.cc:288] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8 [CANDIDATE]: Term 1 election: Requested vote from peers d86646389005421ca729b7c72133d8e7 (127.0.98.1:35997), 9310fb3e9a984873a5c72d2273da0545 (127.0.98.3:43449)
I20260926 11:30:07.618609 5151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8827a39dde394a1fb58bc76933cd46ff" candidate_uuid: "04de65faee334f4ab840af5b889414f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d86646389005421ca729b7c72133d8e7"
I20260926 11:30:07.618616 5413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8827a39dde394a1fb58bc76933cd46ff" candidate_uuid: "04de65faee334f4ab840af5b889414f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9310fb3e9a984873a5c72d2273da0545"
I20260926 11:30:07.618748 5151 raft_consensus.cc:3037] T 8827a39dde394a1fb58bc76933cd46ff P d86646389005421ca729b7c72133d8e7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:07.618752 5413 raft_consensus.cc:3037] T 8827a39dde394a1fb58bc76933cd46ff P 9310fb3e9a984873a5c72d2273da0545 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:07.619845 5413 raft_consensus.cc:2454] T 8827a39dde394a1fb58bc76933cd46ff P 9310fb3e9a984873a5c72d2273da0545 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04de65faee334f4ab840af5b889414f8 in term 1.
I20260926 11:30:07.619858 5151 raft_consensus.cc:2454] T 8827a39dde394a1fb58bc76933cd46ff P d86646389005421ca729b7c72133d8e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04de65faee334f4ab840af5b889414f8 in term 1.
I20260926 11:30:07.620113 5217 leader_election.cc:302] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04de65faee334f4ab840af5b889414f8, 9310fb3e9a984873a5c72d2273da0545; no voters:
I20260926 11:30:07.620280 5484 raft_consensus.cc:2781] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:30:07.620520 5484 raft_consensus.cc:686] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8 [term 1 LEADER]: Becoming Leader. State: Replica: 04de65faee334f4ab840af5b889414f8, State: Running, Role: LEADER
I20260926 11:30:07.620841 5484 consensus_queue.cc:237] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04de65faee334f4ab840af5b889414f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41673 } } peers { permanent_uuid: "d86646389005421ca729b7c72133d8e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35997 } } peers { permanent_uuid: "9310fb3e9a984873a5c72d2273da0545" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43449 } }
I20260926 11:30:07.623881 5008 catalog_manager.cc:5495] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 04de65faee334f4ab840af5b889414f8 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "04de65faee334f4ab840af5b889414f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04de65faee334f4ab840af5b889414f8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d86646389005421ca729b7c72133d8e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9310fb3e9a984873a5c72d2273da0545" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43449 } health_report { overall_health: UNKNOWN } } }
I20260926 11:30:07.635341 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:07.633980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53194 (local address 127.0.98.2:41673)
0926 11:30:07.634083 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:07.634086 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:07.634114 (+ 28us) server_negotiation.cc:407] Connection header received
0926 11:30:07.634261 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:07.634263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:07.634317 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:07.634381 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:07.634984 (+ 603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:07.634987 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:07.634989 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:30:07.634999 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:07.635016 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:30:07.635023 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:07.635090 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:07.635145 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 11:30:07.635175 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 11:30:07.640820 5151 raft_consensus.cc:1261] T 8827a39dde394a1fb58bc76933cd46ff P d86646389005421ca729b7c72133d8e7 [term 1 FOLLOWER]: Refusing update from remote peer 04de65faee334f4ab840af5b889414f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:30:07.640820 5413 raft_consensus.cc:1261] T 8827a39dde394a1fb58bc76933cd46ff P 9310fb3e9a984873a5c72d2273da0545 [term 1 FOLLOWER]: Refusing update from remote peer 04de65faee334f4ab840af5b889414f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:30:07.641505 5484 consensus_queue.cc:1035] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9310fb3e9a984873a5c72d2273da0545" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:30:07.641724 5486 consensus_queue.cc:1035] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d86646389005421ca729b7c72133d8e7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:30:07.659056 5007 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:30:07.659207 5007 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:30:07.661582 5007 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e11f46fa9ba84e749bc854739f7edbe5: Sending DeleteTablet for 3 replicas of tablet 8827a39dde394a1fb58bc76933cd46ff
I20260926 11:30:07.662411 5262 tablet_service.cc:1514] Processing DeleteTablet for tablet 8827a39dde394a1fb58bc76933cd46ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:30:07 UTC) from {username='slave'} at 127.0.0.1:53188
I20260926 11:30:07.662546 5131 tablet_service.cc:1514] Processing DeleteTablet for tablet 8827a39dde394a1fb58bc76933cd46ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:30:07 UTC) from {username='slave'} at 127.0.0.1:46796
I20260926 11:30:07.662583 5393 tablet_service.cc:1514] Processing DeleteTablet for tablet 8827a39dde394a1fb58bc76933cd46ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:30:07 UTC) from {username='slave'} at 127.0.0.1:41728
I20260926 11:30:07.663061 5500 tablet_replica.cc:330] T 8827a39dde394a1fb58bc76933cd46ff P 9310fb3e9a984873a5c72d2273da0545: stopping tablet replica
I20260926 11:30:07.663465 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 5069
I20260926 11:30:07.663477 5500 raft_consensus.cc:2229] T 8827a39dde394a1fb58bc76933cd46ff P 9310fb3e9a984873a5c72d2273da0545 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:30:07.663827 5500 raft_consensus.cc:2258] T 8827a39dde394a1fb58bc76933cd46ff P 9310fb3e9a984873a5c72d2273da0545 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:30:07.664500 5499 tablet_replica.cc:330] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8: stopping tablet replica
I20260926 11:30:07.664776 5499 raft_consensus.cc:2229] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:30:07.665138 5499 raft_consensus.cc:2258] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:30:07.666713 5500 ts_tablet_manager.cc:1905] T 8827a39dde394a1fb58bc76933cd46ff P 9310fb3e9a984873a5c72d2273da0545: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:30:07.669064 5499 ts_tablet_manager.cc:1905] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:30:07.669469 5500 ts_tablet_manager.cc:1918] T 8827a39dde394a1fb58bc76933cd46ff P 9310fb3e9a984873a5c72d2273da0545: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:30:07.669627 5500 log.cc:1196] T 8827a39dde394a1fb58bc76933cd46ff P 9310fb3e9a984873a5c72d2273da0545: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-2/wal/wals/8827a39dde394a1fb58bc76933cd46ff
I20260926 11:30:07.669987 5500 ts_tablet_manager.cc:1939] T 8827a39dde394a1fb58bc76933cd46ff P 9310fb3e9a984873a5c72d2273da0545: Deleting consensus metadata
I20260926 11:30:07.671324 4995 catalog_manager.cc:4841] TS 9310fb3e9a984873a5c72d2273da0545 (127.0.98.3:43449): tablet 8827a39dde394a1fb58bc76933cd46ff (table test-table [id=193320f7b5cc472da252ab56f8a8dd06]) successfully deleted
I20260926 11:30:07.672464 5499 ts_tablet_manager.cc:1918] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:30:07.672566 5499 log.cc:1196] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790422180210734-392-0/minicluster-data/ts-1/wal/wals/8827a39dde394a1fb58bc76933cd46ff
I20260926 11:30:07.672969 5499 ts_tablet_manager.cc:1939] T 8827a39dde394a1fb58bc76933cd46ff P 04de65faee334f4ab840af5b889414f8: Deleting consensus metadata
W20260926 11:30:07.673417 4993 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35997 (error 108)
I20260926 11:30:07.673645 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 5200
I20260926 11:30:07.675596 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:07.675316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35997 (local address 127.0.0.1:46804)
0926 11:30:07.675443 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:07.675527 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 11:30:07.675771 4993 catalog_manager.cc:4584] TS d86646389005421ca729b7c72133d8e7 (127.0.98.1:35997): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8827a39dde394a1fb58bc76933cd46ff: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35997: connect: Connection refused (error 111)
I20260926 11:30:07.679451 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 5331
I20260926 11:30:07.679873 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:07.679469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41673 (local address 127.0.0.1:53196)
0926 11:30:07.679624 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:07.679690 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 11:30:07.684816 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 4978
2026-09-26T11:30:07Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (896 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 11:30:07.705608 392 mini_oidc.cc:152] Starting JWKS server
I20260926 11:30:07.707167 392 webserver.cc:460] Webserver started at https://127.0.0.1:34923/ using document root <none> and password file <none>
I20260926 11:30:07.707351 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 11:30:07.707631 392 webserver.cc:460] Webserver started at http://127.0.0.1:43989/ using document root <none> and password file <none>
2026-09-26T11:30:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:30:07Z Disabled control of system clock
I20260926 11:30:07.716814 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44499
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44499
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34923/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422180210734-392-0/testchainca.cert with env {}
W20260926 11:30:07.823585 5508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:07.823952 5508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:07.824036 5508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:07.828082 5508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:30:07.828166 5508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:07.828193 5508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:07.828235 5508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:30:07.828280 5508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 11:30:07.828312 5508 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 11:30:07.828333 5508 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34923/jwks.json
I20260926 11:30:07.833681 5508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44499
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34923/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422180210734-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:07.834729 5508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:07.835773 5508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:07.844060 5514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:07.844074 5508 server_base.cc:1034] running on GCE node
W20260926 11:30:07.844087 5513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:07.844092 5516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:07.844839 5508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:07.845938 5508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:07.847100 5508 hybrid_clock.cc:648] HybridClock initialized: now 1790422207847085 us; error 27 us; skew 500 ppm
I20260926 11:30:07.849058 5508 webserver.cc:460] Webserver started at http://127.0.98.62:43383/ using document root <none> and password file <none>
I20260926 11:30:07.849649 5508 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:07.849712 5508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:07.849939 5508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:07.852082 5508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "10af331480e74e8a95bd98a88cb7e2d5"
format_stamp: "Formatted at 2026-09-26 11:30:07 on dist-test-slave-mrck"
server_key: "6bb143c6fd90342ca118ad9c780134af"
server_key_iv: "a27a5113bd9ac602af63b8f8e854316f"
server_key_version: "encryptionkey@0"
I20260926 11:30:07.852658 5508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "10af331480e74e8a95bd98a88cb7e2d5"
format_stamp: "Formatted at 2026-09-26 11:30:07 on dist-test-slave-mrck"
server_key: "6bb143c6fd90342ca118ad9c780134af"
server_key_iv: "a27a5113bd9ac602af63b8f8e854316f"
server_key_version: "encryptionkey@0"
I20260926 11:30:07.856227 5508 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:30:07.858624 5522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:07.859899 5508 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:30:07.860029 5508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "10af331480e74e8a95bd98a88cb7e2d5"
format_stamp: "Formatted at 2026-09-26 11:30:07 on dist-test-slave-mrck"
server_key: "6bb143c6fd90342ca118ad9c780134af"
server_key_iv: "a27a5113bd9ac602af63b8f8e854316f"
server_key_version: "encryptionkey@0"
I20260926 11:30:07.860137 5508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:07.902231 5508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:07.902992 5508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:07.903195 5508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:07.910104 5508 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44499
I20260926 11:30:07.910111 5574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44499 every 8 connection(s)
I20260926 11:30:07.911065 5508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:30:07.912320 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 5508
I20260926 11:30:07.912458 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:30:07.912743 392 external_mini_cluster.cc:1428] Setting key 419b69ecd7ba1e068b3287b6522b1e85
I20260926 11:30:07.914628 5575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:07.919886 5575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10af331480e74e8a95bd98a88cb7e2d5: Bootstrap starting.
I20260926 11:30:07.921898 5578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:07.914180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54298 (local address 127.0.98.62:44499)
0926 11:30:07.914892 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:07.914907 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:07.914942 (+ 35us) server_negotiation.cc:407] Connection header received
0926 11:30:07.915578 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:07.915594 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:07.916061 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:07.916388 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:07.916797 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:07.917733 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:07.918651 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:07.919125 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:07.919301 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:07.920225 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:07.920230 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:07.920242 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:30:07.920617 (+ 375us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:07.920671 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:30:07.920690 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:07.920795 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:07.920938 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 11:30:07.921192 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":146,"threads_started":1}
I20260926 11:30:07.922122 5575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10af331480e74e8a95bd98a88cb7e2d5: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:07.922955 5575 log.cc:824] T 00000000000000000000000000000000 P 10af331480e74e8a95bd98a88cb7e2d5: Log is configured to *not* fsync() on all Append() calls
I20260926 11:30:07.924727 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:30:07.925036 5575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10af331480e74e8a95bd98a88cb7e2d5: No bootstrap required, opened a new log
I20260926 11:30:07.928586 5575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10af331480e74e8a95bd98a88cb7e2d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10af331480e74e8a95bd98a88cb7e2d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44499 } }
I20260926 11:30:07.928834 5575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10af331480e74e8a95bd98a88cb7e2d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:07.928915 5575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10af331480e74e8a95bd98a88cb7e2d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10af331480e74e8a95bd98a88cb7e2d5, State: Initialized, Role: FOLLOWER
I20260926 11:30:07.929338 5575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10af331480e74e8a95bd98a88cb7e2d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10af331480e74e8a95bd98a88cb7e2d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44499 } }
I20260926 11:30:07.929456 5575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10af331480e74e8a95bd98a88cb7e2d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:30:07.929495 5575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10af331480e74e8a95bd98a88cb7e2d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:30:07.929561 5575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10af331480e74e8a95bd98a88cb7e2d5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:07.930382 5575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10af331480e74e8a95bd98a88cb7e2d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10af331480e74e8a95bd98a88cb7e2d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44499 } }
I20260926 11:30:07.930661 5575 leader_election.cc:302] T 00000000000000000000000000000000 P 10af331480e74e8a95bd98a88cb7e2d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10af331480e74e8a95bd98a88cb7e2d5; no voters:
I20260926 11:30:07.930919 5575 leader_election.cc:288] T 00000000000000000000000000000000 P 10af331480e74e8a95bd98a88cb7e2d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:30:07.931041 5580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10af331480e74e8a95bd98a88cb7e2d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:30:07.931281 5580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10af331480e74e8a95bd98a88cb7e2d5 [term 1 LEADER]: Becoming Leader. State: Replica: 10af331480e74e8a95bd98a88cb7e2d5, State: Running, Role: LEADER
I20260926 11:30:07.931638 5580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10af331480e74e8a95bd98a88cb7e2d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10af331480e74e8a95bd98a88cb7e2d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44499 } }
I20260926 11:30:07.932263 5575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10af331480e74e8a95bd98a88cb7e2d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:30:07.933466 5582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10af331480e74e8a95bd98a88cb7e2d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10af331480e74e8a95bd98a88cb7e2d5. Latest consensus state: current_term: 1 leader_uuid: "10af331480e74e8a95bd98a88cb7e2d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10af331480e74e8a95bd98a88cb7e2d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44499 } } }
I20260926 11:30:07.933457 5581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10af331480e74e8a95bd98a88cb7e2d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10af331480e74e8a95bd98a88cb7e2d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10af331480e74e8a95bd98a88cb7e2d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44499 } } }
I20260926 11:30:07.933595 5581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10af331480e74e8a95bd98a88cb7e2d5 [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:07.933595 5582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10af331480e74e8a95bd98a88cb7e2d5 [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:07.934077 5589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:30:07.937081 5589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:30:07.942121 5589 catalog_manager.cc:1347] Generated new cluster ID: e00c2d69deb646f0b21128efab09e2cc
I20260926 11:30:07.942191 5589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:30:07.952977 5589 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:30:07.954241 5589 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:30:07.959471 5589 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10af331480e74e8a95bd98a88cb7e2d5: Generated new TSK 0
I20260926 11:30:07.960314 5589 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 11:30:07.997016 5605 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 11:30:07.997135 5605 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 11:30:07.997680 5578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:07.973266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54312 (local address 127.0.98.62:44499)
0926 11:30:07.973436 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:07.973441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:07.973672 (+ 231us) server_negotiation.cc:407] Connection header received
0926 11:30:07.973885 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:07.973888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:07.973934 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:07.974019 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0926 11:30:07.974297 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:07.974821 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:07.976052 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:07.976258 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:07.976344 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:07.976753 (+ 409us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 11:30:07.997321 (+ 20568us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34923/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 11:30:07.997566 (+ 245us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54312: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34923/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":82}
W20260926 11:30:07.997768 5578 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54312: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34923/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 11:30:07.999120 5604 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:07.973200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44499 (local address 127.0.0.1:54312)
0926 11:30:07.973578 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:07.973594 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:07.973731 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:07.974056 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:07.974059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:07.974067 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0926 11:30:07.974199 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:07.974204 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:07.974921 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:07.974925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:07.975940 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:07.975950 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:07.976351 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:07.976378 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:07.976471 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:07.976524 (+ 53us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 11:30:07.997563 (+ 21039us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34923/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 11:30:07.997815 (+ 252us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:44499: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34923/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":134,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:44499: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34923/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 11:30:07.999938 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 5508
2026-09-26T11:30:08Z chronyd exiting
I20260926 11:30:08.037412 392 test_util.cc:175] -----------------------------------------------
I20260926 11:30:08.037496 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790422180210734-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (333 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 11:30:08.038439 392 mini_oidc.cc:152] Starting JWKS server
I20260926 11:30:08.039242 392 webserver.cc:460] Webserver started at https://127.0.0.1:45373/ using document root <none> and password file <none>
I20260926 11:30:08.039327 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 11:30:08.039535 392 webserver.cc:460] Webserver started at http://127.0.0.1:45829/ using document root <none> and password file <none>
2026-09-26T11:30:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:30:08Z Disabled control of system clock
I20260926 11:30:08.048164 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46193
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46193
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45373/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422180210734-392-0/testchainca.cert with env {}
W20260926 11:30:08.158499 5611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:08.158862 5611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:08.158948 5611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:08.163022 5611 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:30:08.163125 5611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:08.163156 5611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:08.163196 5611 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:30:08.163226 5611 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 11:30:08.163264 5611 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 11:30:08.163296 5611 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45373/jwks.json
I20260926 11:30:08.168817 5611 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46193
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45373/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422180210734-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:08.169921 5611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:08.170954 5611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 11:30:08.179445 5611 server_base.cc:1034] running on GCE node
W20260926 11:30:08.179397 5617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:08.179420 5616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:08.179397 5619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:08.180156 5611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:08.181246 5611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:08.182400 5611 hybrid_clock.cc:648] HybridClock initialized: now 1790422208182388 us; error 25 us; skew 500 ppm
I20260926 11:30:08.184291 5611 webserver.cc:460] Webserver started at http://127.0.98.62:35061/ using document root <none> and password file <none>
I20260926 11:30:08.185031 5611 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:08.185103 5611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:08.185499 5611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:08.187937 5611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "f06efe61b46344c58b1cbdfd4e67f362"
format_stamp: "Formatted at 2026-09-26 11:30:08 on dist-test-slave-mrck"
server_key: "07e2468fbd157ced9b5129b35749dcac"
server_key_iv: "f6cf9c3c3cbc3684dbffb76cdba81f79"
server_key_version: "encryptionkey@0"
I20260926 11:30:08.188607 5611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "f06efe61b46344c58b1cbdfd4e67f362"
format_stamp: "Formatted at 2026-09-26 11:30:08 on dist-test-slave-mrck"
server_key: "07e2468fbd157ced9b5129b35749dcac"
server_key_iv: "f6cf9c3c3cbc3684dbffb76cdba81f79"
server_key_version: "encryptionkey@0"
I20260926 11:30:08.192618 5611 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 11:30:08.194964 5625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:08.196244 5611 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:30:08.196393 5611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "f06efe61b46344c58b1cbdfd4e67f362"
format_stamp: "Formatted at 2026-09-26 11:30:08 on dist-test-slave-mrck"
server_key: "07e2468fbd157ced9b5129b35749dcac"
server_key_iv: "f6cf9c3c3cbc3684dbffb76cdba81f79"
server_key_version: "encryptionkey@0"
I20260926 11:30:08.196509 5611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:08.216637 5611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:08.217398 5611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:08.217605 5611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:08.224591 5611 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46193
I20260926 11:30:08.224602 5677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46193 every 8 connection(s)
I20260926 11:30:08.225535 5611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:30:08.228643 5678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:08.233839 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 5611
I20260926 11:30:08.233999 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:30:08.234207 5678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f06efe61b46344c58b1cbdfd4e67f362: Bootstrap starting.
I20260926 11:30:08.234299 392 external_mini_cluster.cc:1428] Setting key 2dc86ca5973f56c7b17b03997d63f686
I20260926 11:30:08.236725 5678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f06efe61b46344c58b1cbdfd4e67f362: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:08.237502 5678 log.cc:824] T 00000000000000000000000000000000 P f06efe61b46344c58b1cbdfd4e67f362: Log is configured to *not* fsync() on all Append() calls
I20260926 11:30:08.239485 5678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f06efe61b46344c58b1cbdfd4e67f362: No bootstrap required, opened a new log
I20260926 11:30:08.243480 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:08.235615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39350 (local address 127.0.98.62:46193)
0926 11:30:08.236284 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:08.236298 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:08.236334 (+ 36us) server_negotiation.cc:407] Connection header received
0926 11:30:08.237045 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:08.237064 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:08.237460 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:08.237776 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:08.238232 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:08.239193 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:08.240398 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:08.240870 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:08.240985 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:08.241788 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:08.241793 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:08.241814 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:30:08.242187 (+ 373us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:08.242239 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:30:08.242257 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:08.242334 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:08.242492 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 11:30:08.242773 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":185,"threads_started":1}
I20260926 11:30:08.243566 5678 raft_consensus.cc:348] T 00000000000000000000000000000000 P f06efe61b46344c58b1cbdfd4e67f362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f06efe61b46344c58b1cbdfd4e67f362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46193 } }
I20260926 11:30:08.243898 5678 raft_consensus.cc:374] T 00000000000000000000000000000000 P f06efe61b46344c58b1cbdfd4e67f362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:08.243978 5678 raft_consensus.cc:729] T 00000000000000000000000000000000 P f06efe61b46344c58b1cbdfd4e67f362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f06efe61b46344c58b1cbdfd4e67f362, State: Initialized, Role: FOLLOWER
I20260926 11:30:08.244406 5678 consensus_queue.cc:260] T 00000000000000000000000000000000 P f06efe61b46344c58b1cbdfd4e67f362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f06efe61b46344c58b1cbdfd4e67f362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46193 } }
I20260926 11:30:08.244552 5678 raft_consensus.cc:388] T 00000000000000000000000000000000 P f06efe61b46344c58b1cbdfd4e67f362 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:30:08.244622 5678 raft_consensus.cc:482] T 00000000000000000000000000000000 P f06efe61b46344c58b1cbdfd4e67f362 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:30:08.244751 5678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f06efe61b46344c58b1cbdfd4e67f362 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:08.245771 5678 raft_consensus.cc:504] T 00000000000000000000000000000000 P f06efe61b46344c58b1cbdfd4e67f362 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f06efe61b46344c58b1cbdfd4e67f362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46193 } }
I20260926 11:30:08.246133 5678 leader_election.cc:302] T 00000000000000000000000000000000 P f06efe61b46344c58b1cbdfd4e67f362 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f06efe61b46344c58b1cbdfd4e67f362; no voters:
I20260926 11:30:08.246482 5678 leader_election.cc:288] T 00000000000000000000000000000000 P f06efe61b46344c58b1cbdfd4e67f362 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:30:08.246522 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:30:08.246536 5683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f06efe61b46344c58b1cbdfd4e67f362 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:30:08.246771 5683 raft_consensus.cc:686] T 00000000000000000000000000000000 P f06efe61b46344c58b1cbdfd4e67f362 [term 1 LEADER]: Becoming Leader. State: Replica: f06efe61b46344c58b1cbdfd4e67f362, State: Running, Role: LEADER
I20260926 11:30:08.247126 5683 consensus_queue.cc:237] T 00000000000000000000000000000000 P f06efe61b46344c58b1cbdfd4e67f362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f06efe61b46344c58b1cbdfd4e67f362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46193 } }
I20260926 11:30:08.247774 5678 sys_catalog.cc:564] T 00000000000000000000000000000000 P f06efe61b46344c58b1cbdfd4e67f362 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:30:08.249082 5684 sys_catalog.cc:455] T 00000000000000000000000000000000 P f06efe61b46344c58b1cbdfd4e67f362 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f06efe61b46344c58b1cbdfd4e67f362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f06efe61b46344c58b1cbdfd4e67f362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46193 } } }
I20260926 11:30:08.249100 5685 sys_catalog.cc:455] T 00000000000000000000000000000000 P f06efe61b46344c58b1cbdfd4e67f362 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f06efe61b46344c58b1cbdfd4e67f362. Latest consensus state: current_term: 1 leader_uuid: "f06efe61b46344c58b1cbdfd4e67f362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f06efe61b46344c58b1cbdfd4e67f362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46193 } } }
I20260926 11:30:08.249270 5685 sys_catalog.cc:458] T 00000000000000000000000000000000 P f06efe61b46344c58b1cbdfd4e67f362 [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:08.249533 5684 sys_catalog.cc:458] T 00000000000000000000000000000000 P f06efe61b46344c58b1cbdfd4e67f362 [sys.catalog]: This master's current role is: LEADER
W20260926 11:30:08.252411 5699 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f06efe61b46344c58b1cbdfd4e67f362: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:30:08.252501 5699 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:30:08.252562 5693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:30:08.253202 5693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:30:08.258585 5693 catalog_manager.cc:1347] Generated new cluster ID: d2e71d381f2647268e6a3e597af71896
I20260926 11:30:08.258651 5693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:30:08.285035 5693 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:30:08.286057 5693 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:30:08.299268 5693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f06efe61b46344c58b1cbdfd4e67f362: Generated new TSK 0
I20260926 11:30:08.300122 5693 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 11:30:08.334762 5708 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 11:30:08.334867 5708 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 11:30:08.335634 5681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:08.319814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39360 (local address 127.0.98.62:46193)
0926 11:30:08.319951 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:08.319956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:08.320116 (+ 160us) server_negotiation.cc:407] Connection header received
0926 11:30:08.320253 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:08.320257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:08.320312 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:08.320428 (+ 116us) server_negotiation.cc:226] Negotiated authn=JWT
0926 11:30:08.320626 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:08.321444 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:08.322283 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:08.322644 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:08.322744 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:08.323033 (+ 289us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 11:30:08.335160 (+ 12127us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45373/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 11:30:08.335484 (+ 324us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39360: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45373/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 11:30:08.335711 5681 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39360: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45373/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 11:30:08.335722 5707 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:08.319669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46193 (local address 127.0.0.1:39360)
0926 11:30:08.320033 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:08.320045 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:08.320136 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:08.320429 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:08.320432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:08.320439 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0926 11:30:08.320526 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:08.320530 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:08.321561 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:08.321563 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:08.322170 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:08.322177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:08.322750 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:08.322768 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:08.322837 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:08.322860 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 11:30:08.335396 (+ 12536us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45373/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 11:30:08.335556 (+ 160us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:46193: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45373/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":146,"threads_started":1}
I20260926 11:30:08.336625 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 5611
2026-09-26T11:30:08Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (336 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 11:30:08.375293 392 mini_oidc.cc:152] Starting JWKS server
I20260926 11:30:08.376212 392 webserver.cc:460] Webserver started at https://127.0.0.1:37699/ using document root <none> and password file <none>
I20260926 11:30:08.376291 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 11:30:08.376507 392 webserver.cc:460] Webserver started at http://127.0.0.1:33035/ using document root <none> and password file <none>
2026-09-26T11:30:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:30:08Z Disabled control of system clock
I20260926 11:30:08.385231 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34521
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34521
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37699/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 11:30:08.493438 5714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:08.493705 5714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:08.493744 5714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:08.497648 5714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:30:08.497730 5714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:08.497792 5714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:08.497843 5714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:30:08.497859 5714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 11:30:08.497875 5714 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 11:30:08.497888 5714 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37699/jwks.json
I20260926 11:30:08.503199 5714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34521
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37699/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:08.504318 5714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:08.505676 5714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:08.514200 5719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:08.514226 5722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:08.514258 5714 server_base.cc:1034] running on GCE node
W20260926 11:30:08.514312 5720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:08.515048 5714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:08.516146 5714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:08.517315 5714 hybrid_clock.cc:648] HybridClock initialized: now 1790422208517304 us; error 24 us; skew 500 ppm
I20260926 11:30:08.519068 5714 webserver.cc:460] Webserver started at http://127.0.98.62:46275/ using document root <none> and password file <none>
I20260926 11:30:08.519635 5714 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:08.519695 5714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:08.519976 5714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:08.521962 5714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "05fbf0d134a74ee199c5b2bc6536471d"
format_stamp: "Formatted at 2026-09-26 11:30:08 on dist-test-slave-mrck"
server_key: "df7f1b852a85b3891d3abe4d8093fd98"
server_key_iv: "31d564d556bb97909403b592c2f37721"
server_key_version: "encryptionkey@0"
I20260926 11:30:08.522451 5714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "05fbf0d134a74ee199c5b2bc6536471d"
format_stamp: "Formatted at 2026-09-26 11:30:08 on dist-test-slave-mrck"
server_key: "df7f1b852a85b3891d3abe4d8093fd98"
server_key_iv: "31d564d556bb97909403b592c2f37721"
server_key_version: "encryptionkey@0"
I20260926 11:30:08.526046 5714 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 11:30:08.528352 5728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:08.529573 5714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:30:08.529697 5714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "05fbf0d134a74ee199c5b2bc6536471d"
format_stamp: "Formatted at 2026-09-26 11:30:08 on dist-test-slave-mrck"
server_key: "df7f1b852a85b3891d3abe4d8093fd98"
server_key_iv: "31d564d556bb97909403b592c2f37721"
server_key_version: "encryptionkey@0"
I20260926 11:30:08.529821 5714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:08.554566 5714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:08.555334 5714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:08.555568 5714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:08.562703 5780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34521 every 8 connection(s)
I20260926 11:30:08.562733 5714 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34521
I20260926 11:30:08.563714 5714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:30:08.566877 5781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:08.571022 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 5714
I20260926 11:30:08.571169 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:30:08.571499 392 external_mini_cluster.cc:1428] Setting key f55531af00af99a337109467aab9d7b2
I20260926 11:30:08.573001 5781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05fbf0d134a74ee199c5b2bc6536471d: Bootstrap starting.
I20260926 11:30:08.575178 5781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05fbf0d134a74ee199c5b2bc6536471d: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:08.575953 5781 log.cc:824] T 00000000000000000000000000000000 P 05fbf0d134a74ee199c5b2bc6536471d: Log is configured to *not* fsync() on all Append() calls
I20260926 11:30:08.577822 5781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05fbf0d134a74ee199c5b2bc6536471d: No bootstrap required, opened a new log
I20260926 11:30:08.580636 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:08.572912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57348 (local address 127.0.98.62:34521)
0926 11:30:08.573531 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:08.573546 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:08.573581 (+ 35us) server_negotiation.cc:407] Connection header received
0926 11:30:08.574192 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:08.574207 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:08.574609 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:08.574946 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:08.575417 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:08.576550 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:08.577422 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:08.577961 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:08.578143 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:08.578938 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:08.578944 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:08.578959 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:30:08.579348 (+ 389us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:08.579401 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:30:08.579421 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:08.579527 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:08.579656 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 11:30:08.579925 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":122,"threads_started":1}
I20260926 11:30:08.581313 5781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05fbf0d134a74ee199c5b2bc6536471d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05fbf0d134a74ee199c5b2bc6536471d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34521 } }
I20260926 11:30:08.581554 5781 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05fbf0d134a74ee199c5b2bc6536471d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:08.581651 5781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05fbf0d134a74ee199c5b2bc6536471d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05fbf0d134a74ee199c5b2bc6536471d, State: Initialized, Role: FOLLOWER
I20260926 11:30:08.582109 5781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05fbf0d134a74ee199c5b2bc6536471d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05fbf0d134a74ee199c5b2bc6536471d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34521 } }
I20260926 11:30:08.582257 5781 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05fbf0d134a74ee199c5b2bc6536471d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:30:08.582347 5781 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05fbf0d134a74ee199c5b2bc6536471d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:30:08.582499 5781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05fbf0d134a74ee199c5b2bc6536471d [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:08.583324 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:30:08.583585 5781 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05fbf0d134a74ee199c5b2bc6536471d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05fbf0d134a74ee199c5b2bc6536471d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34521 } }
I20260926 11:30:08.583923 5781 leader_election.cc:302] T 00000000000000000000000000000000 P 05fbf0d134a74ee199c5b2bc6536471d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05fbf0d134a74ee199c5b2bc6536471d; no voters:
I20260926 11:30:08.584208 5781 leader_election.cc:288] T 00000000000000000000000000000000 P 05fbf0d134a74ee199c5b2bc6536471d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:30:08.584306 5786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05fbf0d134a74ee199c5b2bc6536471d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:30:08.584923 5786 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05fbf0d134a74ee199c5b2bc6536471d [term 1 LEADER]: Becoming Leader. State: Replica: 05fbf0d134a74ee199c5b2bc6536471d, State: Running, Role: LEADER
I20260926 11:30:08.585225 5786 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05fbf0d134a74ee199c5b2bc6536471d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05fbf0d134a74ee199c5b2bc6536471d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34521 } }
I20260926 11:30:08.585381 5781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05fbf0d134a74ee199c5b2bc6536471d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:30:08.586293 5786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05fbf0d134a74ee199c5b2bc6536471d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05fbf0d134a74ee199c5b2bc6536471d. Latest consensus state: current_term: 1 leader_uuid: "05fbf0d134a74ee199c5b2bc6536471d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05fbf0d134a74ee199c5b2bc6536471d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34521 } } }
I20260926 11:30:08.586454 5786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05fbf0d134a74ee199c5b2bc6536471d [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:08.586673 5787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05fbf0d134a74ee199c5b2bc6536471d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05fbf0d134a74ee199c5b2bc6536471d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05fbf0d134a74ee199c5b2bc6536471d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34521 } } }
I20260926 11:30:08.586824 5787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05fbf0d134a74ee199c5b2bc6536471d [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:08.586849 5795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:30:08.589891 5795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:30:08.595091 5795 catalog_manager.cc:1347] Generated new cluster ID: 0d1f9c7958324bf28feee3ff568b36e5
I20260926 11:30:08.595167 5795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:30:08.645418 5795 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:30:08.646725 5795 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:30:08.654906 5795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05fbf0d134a74ee199c5b2bc6536471d: Generated new TSK 0
I20260926 11:30:08.655718 5795 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:30:08.673436 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:08.667461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57360 (local address 127.0.98.62:34521)
0926 11:30:08.667588 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:08.667593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:08.667768 (+ 175us) server_negotiation.cc:407] Connection header received
0926 11:30:08.667982 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:08.667986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:08.668044 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:08.668174 (+ 130us) server_negotiation.cc:226] Negotiated authn=JWT
0926 11:30:08.668400 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:08.669180 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:08.670027 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:08.670365 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:08.670463 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:08.670717 (+ 254us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 11:30:08.672175 (+ 1458us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37699/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 11:30:08.672416 (+ 241us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57360: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37699/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 11:30:08.673511 5784 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57360: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37699/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 11:30:08.674867 5810 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:08.667381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34521 (local address 127.0.0.1:57360)
0926 11:30:08.667665 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:08.667677 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:08.667839 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:08.668175 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:08.668177 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:08.668183 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0926 11:30:08.668301 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:08.668306 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:08.669309 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:08.669312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:08.669921 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:08.669929 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:08.670463 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:08.670478 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:08.670537 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:08.670557 (+ 20us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 11:30:08.672357 (+ 1800us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37699/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 11:30:08.674701 (+ 2344us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34521: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37699/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":2078,"thread_start_us":101,"threads_started":1}
I20260926 11:30:08.675372 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 5714
E20260926 11:30:08.684018 5811 webserver.cc:533] Webserver: sslize failed:
E20260926 11:30:08.684077 5811 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T11:30:08Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (339 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T11:30:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:30:08Z Disabled control of system clock
I20260926 11:30:08.725255 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42593
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42593
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422180210734-392-0/testchainca.cert with env {}
W20260926 11:30:08.832995 5815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:08.833280 5815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:08.833388 5815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:08.837502 5815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:30:08.837567 5815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:08.837582 5815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:08.837599 5815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:30:08.837615 5815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 11:30:08.837677 5815 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 11:30:08.837702 5815 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 11:30:08.843181 5815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42593
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422180210734-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:08.844223 5815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:08.845256 5815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:08.853601 5820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:08.853559 5821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:08.853647 5815 server_base.cc:1034] running on GCE node
W20260926 11:30:08.853777 5823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:08.854285 5815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:08.855261 5815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:08.856426 5815 hybrid_clock.cc:648] HybridClock initialized: now 1790422208856398 us; error 41 us; skew 500 ppm
I20260926 11:30:08.858177 5815 webserver.cc:460] Webserver started at http://127.0.98.62:36723/ using document root <none> and password file <none>
I20260926 11:30:08.858726 5815 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:08.858785 5815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:08.859009 5815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:08.861047 5815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "af6593bc3a62478ba6b26c43e75b8c83"
format_stamp: "Formatted at 2026-09-26 11:30:08 on dist-test-slave-mrck"
server_key: "a5bc0f9ca79a59172ff854c167dece21"
server_key_iv: "5b7fcd5cc13d1a58d1ceaa31099a3644"
server_key_version: "encryptionkey@0"
I20260926 11:30:08.861599 5815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "af6593bc3a62478ba6b26c43e75b8c83"
format_stamp: "Formatted at 2026-09-26 11:30:08 on dist-test-slave-mrck"
server_key: "a5bc0f9ca79a59172ff854c167dece21"
server_key_iv: "5b7fcd5cc13d1a58d1ceaa31099a3644"
server_key_version: "encryptionkey@0"
I20260926 11:30:08.865223 5815 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:30:08.867611 5829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:08.868964 5815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:30:08.869105 5815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "af6593bc3a62478ba6b26c43e75b8c83"
format_stamp: "Formatted at 2026-09-26 11:30:08 on dist-test-slave-mrck"
server_key: "a5bc0f9ca79a59172ff854c167dece21"
server_key_iv: "5b7fcd5cc13d1a58d1ceaa31099a3644"
server_key_version: "encryptionkey@0"
I20260926 11:30:08.869213 5815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:08.908674 5815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:08.909435 5815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:08.909636 5815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:08.916549 5815 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42593
I20260926 11:30:08.916604 5881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42593 every 8 connection(s)
I20260926 11:30:08.917531 5815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:30:08.920727 5882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:08.921689 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 5815
I20260926 11:30:08.921842 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:30:08.922123 392 external_mini_cluster.cc:1428] Setting key 8f9625b68db0733d05d27eeb4df4e40b
I20260926 11:30:08.926723 5882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af6593bc3a62478ba6b26c43e75b8c83: Bootstrap starting.
I20260926 11:30:08.929113 5882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af6593bc3a62478ba6b26c43e75b8c83: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:08.929939 5882 log.cc:824] T 00000000000000000000000000000000 P af6593bc3a62478ba6b26c43e75b8c83: Log is configured to *not* fsync() on all Append() calls
I20260926 11:30:08.931691 5885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:08.923497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48482 (local address 127.0.98.62:42593)
0926 11:30:08.924170 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:08.924186 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:08.924221 (+ 35us) server_negotiation.cc:407] Connection header received
0926 11:30:08.924846 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:08.924863 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:08.925253 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:08.925597 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:08.926033 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:08.927291 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:08.928307 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:08.928772 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:08.928950 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:08.929869 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:08.929874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:08.929888 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:30:08.930350 (+ 462us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:08.930418 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:30:08.930447 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:08.930555 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:08.930711 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 11:30:08.931006 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":159,"threads_started":1}
I20260926 11:30:08.932029 5882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af6593bc3a62478ba6b26c43e75b8c83: No bootstrap required, opened a new log
I20260926 11:30:08.934526 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:30:08.936100 5882 raft_consensus.cc:348] T 00000000000000000000000000000000 P af6593bc3a62478ba6b26c43e75b8c83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af6593bc3a62478ba6b26c43e75b8c83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42593 } }
I20260926 11:30:08.936340 5882 raft_consensus.cc:374] T 00000000000000000000000000000000 P af6593bc3a62478ba6b26c43e75b8c83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:08.936420 5882 raft_consensus.cc:729] T 00000000000000000000000000000000 P af6593bc3a62478ba6b26c43e75b8c83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af6593bc3a62478ba6b26c43e75b8c83, State: Initialized, Role: FOLLOWER
I20260926 11:30:08.936842 5882 consensus_queue.cc:260] T 00000000000000000000000000000000 P af6593bc3a62478ba6b26c43e75b8c83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af6593bc3a62478ba6b26c43e75b8c83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42593 } }
I20260926 11:30:08.936990 5882 raft_consensus.cc:388] T 00000000000000000000000000000000 P af6593bc3a62478ba6b26c43e75b8c83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:30:08.937074 5882 raft_consensus.cc:482] T 00000000000000000000000000000000 P af6593bc3a62478ba6b26c43e75b8c83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:30:08.937187 5882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af6593bc3a62478ba6b26c43e75b8c83 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:08.938097 5882 raft_consensus.cc:504] T 00000000000000000000000000000000 P af6593bc3a62478ba6b26c43e75b8c83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af6593bc3a62478ba6b26c43e75b8c83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42593 } }
I20260926 11:30:08.938457 5882 leader_election.cc:302] T 00000000000000000000000000000000 P af6593bc3a62478ba6b26c43e75b8c83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af6593bc3a62478ba6b26c43e75b8c83; no voters:
I20260926 11:30:08.938822 5882 leader_election.cc:288] T 00000000000000000000000000000000 P af6593bc3a62478ba6b26c43e75b8c83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:30:08.938903 5887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af6593bc3a62478ba6b26c43e75b8c83 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:30:08.939126 5887 raft_consensus.cc:686] T 00000000000000000000000000000000 P af6593bc3a62478ba6b26c43e75b8c83 [term 1 LEADER]: Becoming Leader. State: Replica: af6593bc3a62478ba6b26c43e75b8c83, State: Running, Role: LEADER
I20260926 11:30:08.939448 5887 consensus_queue.cc:237] T 00000000000000000000000000000000 P af6593bc3a62478ba6b26c43e75b8c83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af6593bc3a62478ba6b26c43e75b8c83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42593 } }
I20260926 11:30:08.940068 5882 sys_catalog.cc:564] T 00000000000000000000000000000000 P af6593bc3a62478ba6b26c43e75b8c83 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:30:08.941290 5889 sys_catalog.cc:455] T 00000000000000000000000000000000 P af6593bc3a62478ba6b26c43e75b8c83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af6593bc3a62478ba6b26c43e75b8c83. Latest consensus state: current_term: 1 leader_uuid: "af6593bc3a62478ba6b26c43e75b8c83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af6593bc3a62478ba6b26c43e75b8c83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42593 } } }
I20260926 11:30:08.941269 5888 sys_catalog.cc:455] T 00000000000000000000000000000000 P af6593bc3a62478ba6b26c43e75b8c83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af6593bc3a62478ba6b26c43e75b8c83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af6593bc3a62478ba6b26c43e75b8c83" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42593 } } }
I20260926 11:30:08.941402 5889 sys_catalog.cc:458] T 00000000000000000000000000000000 P af6593bc3a62478ba6b26c43e75b8c83 [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:08.941404 5888 sys_catalog.cc:458] T 00000000000000000000000000000000 P af6593bc3a62478ba6b26c43e75b8c83 [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:08.942432 5896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:30:08.945029 5896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:30:08.950259 5896 catalog_manager.cc:1347] Generated new cluster ID: 0e413d4119ec48afb043d239c200dcdb
I20260926 11:30:08.950323 5896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:30:08.959153 5896 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:30:08.960383 5896 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:30:08.977632 5896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af6593bc3a62478ba6b26c43e75b8c83: Generated new TSK 0
I20260926 11:30:08.978489 5896 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:30:09.100659 5885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:08.993015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48486 (local address 127.0.98.62:42593)
0926 11:30:08.993181 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:08.993186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:08.993359 (+ 173us) server_negotiation.cc:407] Connection header received
0926 11:30:08.993579 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:08.993582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:08.993629 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:08.993728 (+ 99us) server_negotiation.cc:226] Negotiated authn=JWT
0926 11:30:08.993985 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:08.994552 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:08.995793 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:08.996027 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:08.996111 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:08.996471 (+ 360us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 11:30:09.100243 (+103772us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 11:30:09.100536 (+ 293us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48486: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 11:30:09.100765 5885 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48486: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 11:30:09.100864 5911 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:08.992939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42593 (local address 127.0.0.1:48486)
0926 11:30:08.993264 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:08.993279 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:08.993396 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:08.993748 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:08.993752 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:08.993759 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0926 11:30:08.993878 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:08.993884 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:08.994656 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:08.994660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:08.995623 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:08.995632 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:08.996143 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:08.996179 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:08.996268 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:08.996295 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 11:30:09.100500 (+104205us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 11:30:09.100675 (+ 175us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42593: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":123,"threads_started":1}
I20260926 11:30:09.101433 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 5815
2026-09-26T11:30:09Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (405 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (122 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (120 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:30:09 dist-test-slave-mrck krb5kdc[5923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:30:09 dist-test-slave-mrck krb5kdc[5923](info): set up 2 sockets
Sep 26 11:30:09 dist-test-slave-mrck krb5kdc[5923](info): commencing operation
krb5kdc: starting...
W20260926 11:30:10.392163 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:30:10 dist-test-slave-mrck krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:30:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:30:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:30:10.508740 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39973
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39973
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:10.615881 5936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:10.616130 5936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:10.616170 5936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:10.620108 5936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:30:10.620157 5936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:10.620177 5936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:10.620193 5936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:30:10.620208 5936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:30:10.625264 5936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:10.626186 5936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:10.627111 5936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:10.635028 5941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:10.635028 5942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:10.635105 5936 server_base.cc:1034] running on GCE node
W20260926 11:30:10.635089 5944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:10.635874 5936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:10.636873 5936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:10.638025 5936 hybrid_clock.cc:648] HybridClock initialized: now 1790422210637994 us; error 46 us; skew 500 ppm
Sep 26 11:30:10 dist-test-slave-mrck krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:10.640830 5936 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:30:10.641716 5936 webserver.cc:460] Webserver started at http://127.0.98.62:38053/ using document root <none> and password file <none>
I20260926 11:30:10.642261 5936 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:10.642349 5936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:10.642589 5936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:10.644588 5936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "f6d17505dd684f739b8b5e4319e8c8f6"
format_stamp: "Formatted at 2026-09-26 11:30:10 on dist-test-slave-mrck"
server_key: "5419d775b9fbf01967388315f9a3503b"
server_key_iv: "40909cfbd4dcb8817b7958119336ef21"
server_key_version: "encryptionkey@0"
I20260926 11:30:10.645165 5936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "f6d17505dd684f739b8b5e4319e8c8f6"
format_stamp: "Formatted at 2026-09-26 11:30:10 on dist-test-slave-mrck"
server_key: "5419d775b9fbf01967388315f9a3503b"
server_key_iv: "40909cfbd4dcb8817b7958119336ef21"
server_key_version: "encryptionkey@0"
I20260926 11:30:10.648835 5936 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:30:10.651130 5951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:10.652438 5936 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:30:10.652582 5936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "f6d17505dd684f739b8b5e4319e8c8f6"
format_stamp: "Formatted at 2026-09-26 11:30:10 on dist-test-slave-mrck"
server_key: "5419d775b9fbf01967388315f9a3503b"
server_key_iv: "40909cfbd4dcb8817b7958119336ef21"
server_key_version: "encryptionkey@0"
I20260926 11:30:10.652688 5936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:10.665432 5936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:10.668308 5936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:10.668505 5936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:10.675282 5936 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39973
I20260926 11:30:10.675585 6003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39973 every 8 connection(s)
I20260926 11:30:10.676288 5936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:30:10.679227 6004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:10.684399 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 5936
I20260926 11:30:10.684561 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:30:10.684918 392 external_mini_cluster.cc:1428] Setting key 7e33fd5f93d1da334d12a93fd3897a11
I20260926 11:30:10.686077 6004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6d17505dd684f739b8b5e4319e8c8f6: Bootstrap starting.
I20260926 11:30:10.688689 6004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6d17505dd684f739b8b5e4319e8c8f6: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:10.689590 6004 log.cc:824] T 00000000000000000000000000000000 P f6d17505dd684f739b8b5e4319e8c8f6: Log is configured to *not* fsync() on all Append() calls
I20260926 11:30:10.691653 6004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6d17505dd684f739b8b5e4319e8c8f6: No bootstrap required, opened a new log
Sep 26 11:30:10 dist-test-slave-mrck krb5kdc[5923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:30:10.695307 6004 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6d17505dd684f739b8b5e4319e8c8f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d17505dd684f739b8b5e4319e8c8f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39973 } }
I20260926 11:30:10.695514 6004 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6d17505dd684f739b8b5e4319e8c8f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:10.695585 6004 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6d17505dd684f739b8b5e4319e8c8f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d17505dd684f739b8b5e4319e8c8f6, State: Initialized, Role: FOLLOWER
I20260926 11:30:10.696115 6004 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6d17505dd684f739b8b5e4319e8c8f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d17505dd684f739b8b5e4319e8c8f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39973 } }
I20260926 11:30:10.696273 6004 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6d17505dd684f739b8b5e4319e8c8f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:30:10.696341 6004 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6d17505dd684f739b8b5e4319e8c8f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:30:10.696430 6004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6d17505dd684f739b8b5e4319e8c8f6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:10.697515 6004 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6d17505dd684f739b8b5e4319e8c8f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d17505dd684f739b8b5e4319e8c8f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39973 } }
I20260926 11:30:10.697866 6004 leader_election.cc:302] T 00000000000000000000000000000000 P f6d17505dd684f739b8b5e4319e8c8f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6d17505dd684f739b8b5e4319e8c8f6; no voters:
I20260926 11:30:10.698179 6004 leader_election.cc:288] T 00000000000000000000000000000000 P f6d17505dd684f739b8b5e4319e8c8f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:30:10.698259 6009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6d17505dd684f739b8b5e4319e8c8f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:30:10.698565 6009 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6d17505dd684f739b8b5e4319e8c8f6 [term 1 LEADER]: Becoming Leader. State: Replica: f6d17505dd684f739b8b5e4319e8c8f6, State: Running, Role: LEADER
I20260926 11:30:10.698921 6009 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6d17505dd684f739b8b5e4319e8c8f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d17505dd684f739b8b5e4319e8c8f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39973 } }
I20260926 11:30:10.699468 6004 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6d17505dd684f739b8b5e4319e8c8f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:30:10.700075 6009 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6d17505dd684f739b8b5e4319e8c8f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6d17505dd684f739b8b5e4319e8c8f6. Latest consensus state: current_term: 1 leader_uuid: "f6d17505dd684f739b8b5e4319e8c8f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d17505dd684f739b8b5e4319e8c8f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39973 } } }
I20260926 11:30:10.700256 6009 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6d17505dd684f739b8b5e4319e8c8f6 [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:10.700501 6011 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6d17505dd684f739b8b5e4319e8c8f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6d17505dd684f739b8b5e4319e8c8f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6d17505dd684f739b8b5e4319e8c8f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39973 } } }
I20260926 11:30:10.700600 6011 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6d17505dd684f739b8b5e4319e8c8f6 [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:10.700325 6007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:10.686460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54450 (local address 127.0.98.62:39973)
0926 11:30:10.686903 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:10.686917 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:10.686952 (+ 35us) server_negotiation.cc:407] Connection header received
0926 11:30:10.687588 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:10.687605 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:10.688043 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:10.688378 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:10.689175 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:10.690416 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:10.691320 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:10.691934 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:10.692098 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:10.694765 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:10.694771 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:10.694794 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:10.694819 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:10.696887 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:10.697293 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:10.697299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:10.697304 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:10.697448 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:10.697735 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:10.697738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:10.697740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:10.698208 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:10.698447 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:10.699030 (+ 583us) server_negotiation.cc:299] Negotiation successful
0926 11:30:10.699328 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":142,"threads_started":1}
I20260926 11:30:10.700870 6018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:30:10.704509 6018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:30:10.710377 6018 catalog_manager.cc:1347] Generated new cluster ID: e58b93565ce44a0db11323ef2fdf5d38
I20260926 11:30:10.710456 6018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:30:10.725780 6018 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:30:10.727058 6018 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:30:10.734800 6018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6d17505dd684f739b8b5e4319e8c8f6: Generated new TSK 0
I20260926 11:30:10.735634 6018 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:30:10.763868 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39973
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:10.874454 6030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:10.874742 6030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:10.874819 6030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:10.879004 6030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:10.879105 6030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:10.879200 6030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:30:10.884648 6030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:10.885830 6030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:10.886914 6030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:10.896456 6036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:10.896503 6038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:10.896467 6035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:10.896862 6030 server_base.cc:1034] running on GCE node
I20260926 11:30:10.897269 6030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:10.897946 6030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:10.899118 6030 hybrid_clock.cc:648] HybridClock initialized: now 1790422210899060 us; error 64 us; skew 500 ppm
Sep 26 11:30:10 dist-test-slave-mrck krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:10.907562 6030 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:30:10.908617 6030 webserver.cc:460] Webserver started at http://127.0.98.1:36793/ using document root <none> and password file <none>
I20260926 11:30:10.909202 6030 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:10.909294 6030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:10.909536 6030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:10.911582 6030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-0/data/instance:
uuid: "59ee01cb42a4448aab594b7774f2c606"
format_stamp: "Formatted at 2026-09-26 11:30:10 on dist-test-slave-mrck"
server_key: "a6b7b28a654fdf5047d9dec258fd3aea"
server_key_iv: "4821896c1f76dd6f236c6e511a76323e"
server_key_version: "encryptionkey@0"
I20260926 11:30:10.912220 6030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "59ee01cb42a4448aab594b7774f2c606"
format_stamp: "Formatted at 2026-09-26 11:30:10 on dist-test-slave-mrck"
server_key: "a6b7b28a654fdf5047d9dec258fd3aea"
server_key_iv: "4821896c1f76dd6f236c6e511a76323e"
server_key_version: "encryptionkey@0"
I20260926 11:30:10.915937 6030 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:30:10.918301 6045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:10.919459 6030 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:30:10.919620 6030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "59ee01cb42a4448aab594b7774f2c606"
format_stamp: "Formatted at 2026-09-26 11:30:10 on dist-test-slave-mrck"
server_key: "a6b7b28a654fdf5047d9dec258fd3aea"
server_key_iv: "4821896c1f76dd6f236c6e511a76323e"
server_key_version: "encryptionkey@0"
I20260926 11:30:10.919780 6030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:10.936364 6030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:10.939678 6030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:10.939931 6030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:10.940519 6030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:10.941303 6030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:10.941378 6030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:10.941461 6030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:10.941511 6030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:10.950069 6030 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44927
I20260926 11:30:10.950115 6158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44927 every 8 connection(s)
I20260926 11:30:10.950923 6030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:30:10.960381 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 6030
I20260926 11:30:10.960477 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-0/wal/instance
Sep 26 11:30:10 dist-test-slave-mrck krb5kdc[5923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:30:10.960738 392 external_mini_cluster.cc:1428] Setting key 8c9d98a04f65f57a6df3f4e872d710c0
I20260926 11:30:10.965737 6007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:10.952661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44413 (local address 127.0.98.62:39973)
0926 11:30:10.952789 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:10.952793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:10.953679 (+ 886us) server_negotiation.cc:407] Connection header received
0926 11:30:10.955032 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:10.955036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:10.955098 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:10.955176 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:10.956552 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:10.957163 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:10.958288 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:10.958528 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:10.958640 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:10.961382 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:10.961385 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:10.961387 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:10.961412 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:10.962906 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:10.963690 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:10.963693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:10.963694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:10.963845 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:10.964424 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:10.964430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:10.964435 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:10.964700 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:10.964821 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:10.965368 (+ 547us) server_negotiation.cc:299] Negotiation successful
0926 11:30:10.965508 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 11:30:10.966602 6161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:10.953069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39973 (local address 127.0.98.1:44413)
0926 11:30:10.953536 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:10.953577 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:10.954824 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:10.955336 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:10.955368 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:10.955833 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:10.956383 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:10.956406 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:10.957291 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:10.957295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:10.958161 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:10.958169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:10.958675 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:10.958679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:10.958859 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:10.959453 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:10.959469 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:10.961175 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:10.963073 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:10.963083 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:10.963102 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:10.963553 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:10.963945 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:10.963952 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:10.963958 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:10.964182 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:10.964821 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:10.964826 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:10.965106 (+ 280us) client_negotiation.cc:769] Sending connection context
0926 11:30:10.965310 (+ 204us) client_negotiation.cc:240] Negotiation successful
0926 11:30:10.965631 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":126,"threads_started":1}
I20260926 11:30:10.968490 6159 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39973
I20260926 11:30:10.968791 6159 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:10.969331 6159 heartbeater.cc:511] Master 127.0.98.62:39973 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 11:30:10.970948 5968 ts_manager.cc:194] Registered new tserver with Master: 59ee01cb42a4448aab594b7774f2c606 (127.0.98.1:44927)
I20260926 11:30:10.972260 5968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44413
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:30:10.988847 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39973
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:11.098701 6164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:11.098948 6164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:11.098989 6164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:11.103482 6164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:11.103583 6164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:11.103684 6164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:30:11.109052 6164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:11.110067 6164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:11.111008 6164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:11.120366 6170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:11.120494 6169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:11.120530 6172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:11.120630 6164 server_base.cc:1034] running on GCE node
I20260926 11:30:11.121038 6164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:11.121662 6164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:11.122826 6164 hybrid_clock.cc:648] HybridClock initialized: now 1790422211122814 us; error 32 us; skew 500 ppm
Sep 26 11:30:11 dist-test-slave-mrck krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:11.125612 6164 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:30:11.126493 6164 webserver.cc:460] Webserver started at http://127.0.98.2:39533/ using document root <none> and password file <none>
I20260926 11:30:11.127023 6164 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:11.127084 6164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:11.127244 6164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:11.129294 6164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-1/data/instance:
uuid: "85668e854e1f42a8854a7bbce638ee95"
format_stamp: "Formatted at 2026-09-26 11:30:11 on dist-test-slave-mrck"
server_key: "3c769f95ae241be13f4fbc16b0cfc956"
server_key_iv: "d8b6e484ef724e7bb3ac3169b45ebdea"
server_key_version: "encryptionkey@0"
I20260926 11:30:11.129813 6164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "85668e854e1f42a8854a7bbce638ee95"
format_stamp: "Formatted at 2026-09-26 11:30:11 on dist-test-slave-mrck"
server_key: "3c769f95ae241be13f4fbc16b0cfc956"
server_key_iv: "d8b6e484ef724e7bb3ac3169b45ebdea"
server_key_version: "encryptionkey@0"
I20260926 11:30:11.133441 6164 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 11:30:11.135675 6179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:11.136904 6164 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:30:11.136997 6164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "85668e854e1f42a8854a7bbce638ee95"
format_stamp: "Formatted at 2026-09-26 11:30:11 on dist-test-slave-mrck"
server_key: "3c769f95ae241be13f4fbc16b0cfc956"
server_key_iv: "d8b6e484ef724e7bb3ac3169b45ebdea"
server_key_version: "encryptionkey@0"
I20260926 11:30:11.137074 6164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:11.155643 6164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:11.159093 6164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:11.159245 6164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:11.159788 6164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:11.160537 6164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:11.160580 6164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:11.160617 6164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:11.160640 6164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:11.169688 6164 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34955
I20260926 11:30:11.169708 6292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34955 every 8 connection(s)
I20260926 11:30:11.170578 6164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:30:11.174818 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 6164
I20260926 11:30:11.174947 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:30:11.175236 392 external_mini_cluster.cc:1428] Setting key 165cb5bf840e31cb1565963c9ae5e37c
Sep 26 11:30:11 dist-test-slave-mrck krb5kdc[5923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 11:30:11.185701 6007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:11.172408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39367 (local address 127.0.98.62:39973)
0926 11:30:11.172563 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:11.172566 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:11.173421 (+ 855us) server_negotiation.cc:407] Connection header received
0926 11:30:11.174793 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:11.174810 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:11.174871 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:11.174964 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:11.176708 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:11.177217 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:11.178765 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:11.179153 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:11.179278 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:11.181811 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:11.181813 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:11.181815 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:11.181837 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:11.183411 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:11.184081 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:11.184084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:11.184085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:11.184162 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:11.184556 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:11.184558 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:11.184559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:11.184722 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:11.184814 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:11.185429 (+ 615us) server_negotiation.cc:299] Negotiation successful
0926 11:30:11.185585 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 11:30:11.186618 6295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:11.172798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39973 (local address 127.0.98.2:39367)
0926 11:30:11.173241 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:11.173284 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:11.174579 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:11.175221 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:11.175229 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:11.175776 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:11.176391 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:11.176405 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:11.177345 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:11.177349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:11.178649 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:11.178658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:11.179294 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:11.179297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:11.179432 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:11.180011 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:11.180036 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:11.181657 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:11.183539 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:11.183545 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:11.183561 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:11.183979 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:11.184261 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:11.184265 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:11.184269 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:11.184437 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:11.184858 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:11.184868 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:11.185188 (+ 320us) client_negotiation.cc:769] Sending connection context
0926 11:30:11.185378 (+ 190us) client_negotiation.cc:240] Negotiation successful
0926 11:30:11.185724 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":98,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 11:30:11.187825 6293 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39973
I20260926 11:30:11.188083 6293 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:11.188611 6293 heartbeater.cc:511] Master 127.0.98.62:39973 requested a full tablet report, sending...
I20260926 11:30:11.189612 5968 ts_manager.cc:194] Registered new tserver with Master: 85668e854e1f42a8854a7bbce638ee95 (127.0.98.2:34955)
I20260926 11:30:11.190171 5968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39367
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:30:11.202416 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39973
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:11.314183 6298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:11.314468 6298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:11.314512 6298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:11.318969 6298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:11.319033 6298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:11.319095 6298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:30:11.324981 6298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:11.326129 6298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:11.327221 6298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:11.336956 6303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:11.336974 6306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:11.336973 6304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:11.337193 6298 server_base.cc:1034] running on GCE node
I20260926 11:30:11.337773 6298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:11.338465 6298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:11.339635 6298 hybrid_clock.cc:648] HybridClock initialized: now 1790422211339620 us; error 32 us; skew 500 ppm
Sep 26 11:30:11 dist-test-slave-mrck krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:11.342628 6298 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:30:11.343495 6298 webserver.cc:460] Webserver started at http://127.0.98.3:35489/ using document root <none> and password file <none>
I20260926 11:30:11.344072 6298 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:11.344134 6298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:11.344300 6298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:11.346246 6298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-2/data/instance:
uuid: "5d248cdc6bf848e7a310154155f0d9b8"
format_stamp: "Formatted at 2026-09-26 11:30:11 on dist-test-slave-mrck"
server_key: "d056bcb996f9dcc03a957f4ae7b1379e"
server_key_iv: "d4d42b6ece63a448f431615397e810fb"
server_key_version: "encryptionkey@0"
I20260926 11:30:11.346762 6298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5d248cdc6bf848e7a310154155f0d9b8"
format_stamp: "Formatted at 2026-09-26 11:30:11 on dist-test-slave-mrck"
server_key: "d056bcb996f9dcc03a957f4ae7b1379e"
server_key_iv: "d4d42b6ece63a448f431615397e810fb"
server_key_version: "encryptionkey@0"
I20260926 11:30:11.350483 6298 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260926 11:30:11.353029 6313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:11.354369 6298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 11:30:11.354485 6298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "5d248cdc6bf848e7a310154155f0d9b8"
format_stamp: "Formatted at 2026-09-26 11:30:11 on dist-test-slave-mrck"
server_key: "d056bcb996f9dcc03a957f4ae7b1379e"
server_key_iv: "d4d42b6ece63a448f431615397e810fb"
server_key_version: "encryptionkey@0"
I20260926 11:30:11.354629 6298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:11.373746 6298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:11.377113 6298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:11.377313 6298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:11.377913 6298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:11.378720 6298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:11.378765 6298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:11.378829 6298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:11.378865 6298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:11.387399 6298 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40909
I20260926 11:30:11.387480 6426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40909 every 8 connection(s)
I20260926 11:30:11.388379 6298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:30:11.388818 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 6298
I20260926 11:30:11.388957 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:30:11.389313 392 external_mini_cluster.cc:1428] Setting key fa7c9693bcd3f6ea10bf5560cd9b1db4
Sep 26 11:30:11 dist-test-slave-mrck krb5kdc[5923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:30:11.403481 6007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:11.390394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60025 (local address 127.0.98.62:39973)
0926 11:30:11.390538 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:11.390544 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:11.391502 (+ 958us) server_negotiation.cc:407] Connection header received
0926 11:30:11.393106 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:11.393110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:11.393169 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:11.393251 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:11.394468 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:11.395020 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:11.396323 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:11.396563 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:11.396684 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:11.399396 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:11.399399 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:11.399401 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:11.399426 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:11.400979 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:11.401584 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:11.401587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:11.401588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:11.401685 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:11.402056 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:11.402059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:11.402065 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:11.402328 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:11.402427 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:11.403158 (+ 731us) server_negotiation.cc:299] Negotiation successful
0926 11:30:11.403291 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 11:30:11.404318 6429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:11.390829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39973 (local address 127.0.98.3:60025)
0926 11:30:11.391337 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:11.391411 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:11.392888 (+ 1477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:11.393434 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:11.393442 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:11.393812 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:11.394310 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:11.394324 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:11.395155 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:11.395159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:11.396170 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:11.396180 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:11.396721 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:11.396727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:11.396911 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:11.397626 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:11.397644 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:11.399190 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:11.401155 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:11.401161 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:11.401174 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:11.401460 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:11.401822 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:11.401824 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:11.401825 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:11.401959 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:11.402423 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:11.402430 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:11.402761 (+ 331us) client_negotiation.cc:769] Sending connection context
0926 11:30:11.402984 (+ 223us) client_negotiation.cc:240] Negotiation successful
0926 11:30:11.403335 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":121,"threads_started":1}
I20260926 11:30:11.405426 6427 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39973
I20260926 11:30:11.405650 6427 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:11.406123 6427 heartbeater.cc:511] Master 127.0.98.62:39973 requested a full tablet report, sending...
I20260926 11:30:11.407140 5968 ts_manager.cc:194] Registered new tserver with Master: 5d248cdc6bf848e7a310154155f0d9b8 (127.0.98.3:40909)
I20260926 11:30:11.407724 5968 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60025
I20260926 11:30:11.416029 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/test-admin.keytab.
Sep 26 11:30:11 dist-test-slave-mrck krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:11.433211 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 11:30:11.436143 6434 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:11.434656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35512 (local address 127.0.98.1:44927)
0926 11:30:11.434914 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:11.434928 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:11.434952 (+ 24us) server_negotiation.cc:407] Connection header received
0926 11:30:11.435025 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:11.435031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:11.435222 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:11.435356 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:11.435949 (+ 593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35512: BlockingRecv error: recv got EOF from 127.0.0.1:35512 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":67,"threads_started":1}
W20260926 11:30:11.436133 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/krb5cc' of type FILE
Sep 26 11:30:11 dist-test-slave-mrck krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:11.437407 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 11:30:11 dist-test-slave-mrck krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:11.438465 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 11:30:11 dist-test-slave-mrck krb5kdc[5923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 11:30:11.448383 6434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:11.439414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35520 (local address 127.0.98.1:44927)
0926 11:30:11.439545 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:11.439550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:11.439669 (+ 119us) server_negotiation.cc:407] Connection header received
0926 11:30:11.439855 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:11.439858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:11.439909 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:11.439978 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:11.440506 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:11.441362 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:11.442142 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:11.442520 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:11.442601 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:11.445008 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:11.445017 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:11.445032 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:11.445056 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:11.446597 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:11.447015 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:11.447021 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:11.447022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:11.447095 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:11.447355 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:11.447358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:11.447359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:11.447598 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:11.447818 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:11.448031 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 11:30:11.448201 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 11:30:11.452214 6434 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:11.451240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35526 (local address 127.0.98.1:44927)
0926 11:30:11.451381 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:11.451384 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:11.451398 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:30:11.451514 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:11.451516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:11.451551 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:11.451614 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:11.452087 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35526: BlockingRecv error: recv got EOF from 127.0.0.1:35526 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 11:30:11.452266 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/krb5cc' of type FILE
Sep 26 11:30:11 dist-test-slave-mrck krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:11.453476 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 11:30:11 dist-test-slave-mrck krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:11.454356 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 11:30:11 dist-test-slave-mrck krb5kdc[5923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 11:30:11.464188 6434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:11.455133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35536 (local address 127.0.98.1:44927)
0926 11:30:11.455322 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:11.455336 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:11.455351 (+ 15us) server_negotiation.cc:407] Connection header received
0926 11:30:11.455543 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:11.455547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:11.455589 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:11.455681 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:11.456379 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:11.457286 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:11.458083 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:11.458332 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:11.458548 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:11.461039 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:11.461043 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:11.461046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:11.461082 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:11.462451 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:11.462916 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:11.462920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:11.462922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:11.463006 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:11.463292 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:11.463297 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:11.463298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:11.463490 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:11.463627 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:11.463909 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 11:30:11.464025 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 11:30:11.466923 6434 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:11.466017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35544 (local address 127.0.98.1:44927)
0926 11:30:11.466149 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:11.466152 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:11.466184 (+ 32us) server_negotiation.cc:407] Connection header received
0926 11:30:11.466331 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:11.466334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:11.466374 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:11.466447 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:11.466824 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35544: BlockingRecv error: recv got EOF from 127.0.0.1:35544 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 11:30:11.467043 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790422180210734-392-0/krb5kdc/krb5cc' of type FILE
Sep 26 11:30:11 dist-test-slave-mrck krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:11.468137 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 11:30:11 dist-test-slave-mrck krb5kdc[5923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:11.469084 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 11:30:11 dist-test-slave-mrck krb5kdc[5923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 11:30:11.478261 6434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:11.469942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35554 (local address 127.0.98.1:44927)
0926 11:30:11.470078 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:11.470083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:11.470191 (+ 108us) server_negotiation.cc:407] Connection header received
0926 11:30:11.470308 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:11.470311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:11.470373 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:11.470460 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:11.470960 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:11.471465 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:11.472376 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:11.472601 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:11.472709 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:11.475125 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:11.475128 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:11.475131 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:11.475154 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:11.476641 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:11.477042 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:11.477046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:11.477047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:11.477138 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:11.477408 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:11.477410 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:11.477412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:11.477594 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:11.477720 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:11.477895 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 11:30:11.478028 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 11:30:11.479365 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 6030
I20260926 11:30:11.485682 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 6164
I20260926 11:30:11.491140 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 6298
I20260926 11:30:11.496634 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 5936
2026-09-26T11:30:11Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2153 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:30:11 dist-test-slave-mrck krb5kdc[6447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:30:11 dist-test-slave-mrck krb5kdc[6447](info): set up 2 sockets
Sep 26 11:30:11 dist-test-slave-mrck krb5kdc[6447](info): commencing operation
krb5kdc: starting...
W20260926 11:30:12.536227 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:30:12 dist-test-slave-mrck krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:30:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:30:12Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260926 11:30:12.654948 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33195
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33195
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:12.765956 6460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:12.766288 6460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:12.766369 6460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:12.770500 6460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:30:12.770598 6460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:12.770634 6460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:12.770676 6460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:30:12.770712 6460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:30:12.776046 6460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:12.777299 6460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:12.778378 6460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:12.786736 6468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:12.786755 6460 server_base.cc:1034] running on GCE node
W20260926 11:30:12.786716 6465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:12.786712 6466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:12.787552 6460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:12.788793 6460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:12.789958 6460 hybrid_clock.cc:648] HybridClock initialized: now 1790422212789945 us; error 27 us; skew 500 ppm
Sep 26 11:30:12 dist-test-slave-mrck krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422212, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:12.792708 6460 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260926 11:30:12.793573 6460 webserver.cc:460] Webserver started at http://127.0.98.62:39817/ using document root <none> and password file <none>
I20260926 11:30:12.794076 6460 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:12.794133 6460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:12.794296 6460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:12.796293 6460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "f281a9b384774f93b3ca169c35839d66"
format_stamp: "Formatted at 2026-09-26 11:30:12 on dist-test-slave-mrck"
server_key: "eb9599ea863a8b417d35298da3c8d2bf"
server_key_iv: "7006699595aba0a5c2cfe68991951f23"
server_key_version: "encryptionkey@0"
I20260926 11:30:12.796803 6460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "f281a9b384774f93b3ca169c35839d66"
format_stamp: "Formatted at 2026-09-26 11:30:12 on dist-test-slave-mrck"
server_key: "eb9599ea863a8b417d35298da3c8d2bf"
server_key_iv: "7006699595aba0a5c2cfe68991951f23"
server_key_version: "encryptionkey@0"
I20260926 11:30:12.800297 6460 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:30:12.802627 6475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:12.803972 6460 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:30:12.804098 6460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "f281a9b384774f93b3ca169c35839d66"
format_stamp: "Formatted at 2026-09-26 11:30:12 on dist-test-slave-mrck"
server_key: "eb9599ea863a8b417d35298da3c8d2bf"
server_key_iv: "7006699595aba0a5c2cfe68991951f23"
server_key_version: "encryptionkey@0"
I20260926 11:30:12.804207 6460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:12.846213 6460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:12.849713 6460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:12.849923 6460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:12.857090 6460 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33195
I20260926 11:30:12.857383 6527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33195 every 8 connection(s)
I20260926 11:30:12.858084 6460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:30:12.860884 6528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:12.861438 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 6460
I20260926 11:30:12.861572 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:30:12.861852 392 external_mini_cluster.cc:1428] Setting key c1bfb3c0ac10a16b571f03a789e2f895
I20260926 11:30:12.867139 6528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66: Bootstrap starting.
I20260926 11:30:12.869797 6528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:12.870745 6528 log.cc:824] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66: Log is configured to *not* fsync() on all Append() calls
Sep 26 11:30:12 dist-test-slave-mrck krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 11:30:12.872726 6528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66: No bootstrap required, opened a new log
I20260926 11:30:12.876607 6528 raft_consensus.cc:348] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } }
I20260926 11:30:12.876861 6528 raft_consensus.cc:374] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:12.876941 6528 raft_consensus.cc:729] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f281a9b384774f93b3ca169c35839d66, State: Initialized, Role: FOLLOWER
I20260926 11:30:12.876942 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:12.863277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59736 (local address 127.0.98.62:33195)
0926 11:30:12.863783 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:12.863805 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:12.863850 (+ 45us) server_negotiation.cc:407] Connection header received
0926 11:30:12.864499 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:12.864517 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:12.864871 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:12.865163 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:12.866087 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:12.867091 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:12.868239 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:12.868776 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:12.868910 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:12.871969 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:12.871974 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:12.871987 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:12.872012 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:12.873731 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:12.874187 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:12.874197 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:12.874202 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:12.874349 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:12.874716 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:12.874720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:12.874722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:12.875351 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:12.875565 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:12.875826 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 11:30:12.876144 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":142,"threads_started":1}
I20260926 11:30:12.877489 6528 consensus_queue.cc:260] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } }
I20260926 11:30:12.877646 6528 raft_consensus.cc:388] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:30:12.877707 6528 raft_consensus.cc:482] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:30:12.877804 6528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:12.878731 6528 raft_consensus.cc:504] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } }
I20260926 11:30:12.879099 6528 leader_election.cc:302] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f281a9b384774f93b3ca169c35839d66; no voters:
I20260926 11:30:12.879427 6528 leader_election.cc:288] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:30:12.879555 6533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:30:12.879808 6533 raft_consensus.cc:686] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [term 1 LEADER]: Becoming Leader. State: Replica: f281a9b384774f93b3ca169c35839d66, State: Running, Role: LEADER
I20260926 11:30:12.880148 6533 consensus_queue.cc:237] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } }
I20260926 11:30:12.880640 6528 sys_catalog.cc:564] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:30:12.881814 6535 sys_catalog.cc:455] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f281a9b384774f93b3ca169c35839d66. Latest consensus state: current_term: 1 leader_uuid: "f281a9b384774f93b3ca169c35839d66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } }
I20260926 11:30:12.881866 6534 sys_catalog.cc:455] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f281a9b384774f93b3ca169c35839d66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } }
I20260926 11:30:12.881924 6535 sys_catalog.cc:458] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:12.881955 6534 sys_catalog.cc:458] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:12.882285 6542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:30:12.885427 6542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:30:12.890894 6542 catalog_manager.cc:1347] Generated new cluster ID: 7d38a3fb7f064528b10e3eedc9ccf1ce
I20260926 11:30:12.890972 6542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:30:12.907766 6542 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:30:12.908975 6542 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:30:12.918936 6542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66: Generated new TSK 0
I20260926 11:30:12.919852 6542 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260926 11:30:12.954013 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33195
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:13.064625 6554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:13.064900 6554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:13.064949 6554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:13.069018 6554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:13.069073 6554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:13.069132 6554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:30:13.074327 6554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:13.075366 6554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:13.076426 6554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:13.086335 6560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:13.086359 6559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:13.086416 6562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:13.086736 6554 server_base.cc:1034] running on GCE node
I20260926 11:30:13.087159 6554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:13.087929 6554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:13.089136 6554 hybrid_clock.cc:648] HybridClock initialized: now 1790422213089113 us; error 37 us; skew 500 ppm
Sep 26 11:30:13 dist-test-slave-mrck krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422213, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:13.092724 6554 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 11:30:13.093631 6554 webserver.cc:460] Webserver started at http://127.0.98.1:45105/ using document root <none> and password file <none>
I20260926 11:30:13.094193 6554 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:13.094252 6554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:13.094491 6554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:13.096602 6554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-0/data/instance:
uuid: "4b3cc5c906eb42cca07bc2854fc9ff3c"
format_stamp: "Formatted at 2026-09-26 11:30:13 on dist-test-slave-mrck"
server_key: "ce329de21b58ba6969f98b1b4b6ca8e0"
server_key_iv: "5540601aaefd903bc46b61df1643990f"
server_key_version: "encryptionkey@0"
I20260926 11:30:13.097164 6554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4b3cc5c906eb42cca07bc2854fc9ff3c"
format_stamp: "Formatted at 2026-09-26 11:30:13 on dist-test-slave-mrck"
server_key: "ce329de21b58ba6969f98b1b4b6ca8e0"
server_key_iv: "5540601aaefd903bc46b61df1643990f"
server_key_version: "encryptionkey@0"
I20260926 11:30:13.100935 6554 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260926 11:30:13.103431 6569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:13.104818 6554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 11:30:13.104972 6554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "4b3cc5c906eb42cca07bc2854fc9ff3c"
format_stamp: "Formatted at 2026-09-26 11:30:13 on dist-test-slave-mrck"
server_key: "ce329de21b58ba6969f98b1b4b6ca8e0"
server_key_iv: "5540601aaefd903bc46b61df1643990f"
server_key_version: "encryptionkey@0"
I20260926 11:30:13.105085 6554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:13.127513 6554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:13.130900 6554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:13.131104 6554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:13.131701 6554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:13.132556 6554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:13.132632 6554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:13.132706 6554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:13.132761 6554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:13.141238 6554 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46203
I20260926 11:30:13.141283 6682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46203 every 8 connection(s)
I20260926 11:30:13.142100 6554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:30:13.150460 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 6554
I20260926 11:30:13.150614 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:30:13.150903 392 external_mini_cluster.cc:1428] Setting key e418b7c83172904343d3a131614682ca
Sep 26 11:30:13 dist-test-slave-mrck krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422213, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 11:30:13.156270 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:13.143886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52835 (local address 127.0.98.62:33195)
0926 11:30:13.144020 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:13.144028 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:13.145216 (+ 1188us) server_negotiation.cc:407] Connection header received
0926 11:30:13.146107 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:13.146110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:13.146165 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:13.146278 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:13.147395 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.148009 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.149122 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.149376 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.149487 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:13.152198 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:13.152201 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:13.152204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:13.152232 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:13.154051 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:13.154695 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:13.154697 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:13.154699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:13.154794 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:13.155161 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:13.155163 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:13.155164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:13.155319 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:13.155390 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:13.156046 (+ 656us) server_negotiation.cc:299] Negotiation successful
0926 11:30:13.156142 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 11:30:13.157449 6685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:13.144239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.98.1:52835)
0926 11:30:13.145071 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:13.145113 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:13.145908 (+ 795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:13.146442 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:13.146448 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:13.146800 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:13.147253 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:13.147267 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.148128 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.148132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:13.149000 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:13.149008 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.149526 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.149530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:13.149697 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:13.150298 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:13.150313 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:13.152042 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:13.154187 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:13.154195 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:13.154212 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:13.154588 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:13.154881 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:13.154887 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:13.154890 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:13.155066 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:13.155423 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:13.155432 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:13.155790 (+ 358us) client_negotiation.cc:769] Sending connection context
0926 11:30:13.156014 (+ 224us) client_negotiation.cc:240] Negotiation successful
0926 11:30:13.156366 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":526,"threads_started":1}
I20260926 11:30:13.158816 6683 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33195
I20260926 11:30:13.159053 6683 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:13.159512 6683 heartbeater.cc:511] Master 127.0.98.62:33195 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 11:30:13.161146 6492 ts_manager.cc:194] Registered new tserver with Master: 4b3cc5c906eb42cca07bc2854fc9ff3c (127.0.98.1:46203)
I20260926 11:30:13.162473 6492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52835
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260926 11:30:13.178360 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33195
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:13.286551 6688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:13.286796 6688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:13.286840 6688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:13.290738 6688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:13.290793 6688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:13.290851 6688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:30:13.296032 6688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:13.297019 6688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:13.297962 6688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:13.307502 6693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:13.307545 6694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:13.307557 6696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:13.307776 6688 server_base.cc:1034] running on GCE node
I20260926 11:30:13.308338 6688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:13.308980 6688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:13.310158 6688 hybrid_clock.cc:648] HybridClock initialized: now 1790422213310126 us; error 50 us; skew 500 ppm
Sep 26 11:30:13 dist-test-slave-mrck krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422213, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:13.312865 6688 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 11:30:13.313768 6688 webserver.cc:460] Webserver started at http://127.0.98.2:44247/ using document root <none> and password file <none>
I20260926 11:30:13.314368 6688 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:13.314457 6688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:13.314684 6688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:13.316728 6688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/data/instance:
uuid: "26d2d13738244b0399f95e5e8a836d82"
format_stamp: "Formatted at 2026-09-26 11:30:13 on dist-test-slave-mrck"
server_key: "6f2e6ede0f233ce89926bb49793a0621"
server_key_iv: "4b8808bf78cae1ed32dab590f8cd54b5"
server_key_version: "encryptionkey@0"
I20260926 11:30:13.317278 6688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "26d2d13738244b0399f95e5e8a836d82"
format_stamp: "Formatted at 2026-09-26 11:30:13 on dist-test-slave-mrck"
server_key: "6f2e6ede0f233ce89926bb49793a0621"
server_key_iv: "4b8808bf78cae1ed32dab590f8cd54b5"
server_key_version: "encryptionkey@0"
I20260926 11:30:13.320796 6688 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260926 11:30:13.323062 6703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:13.324302 6688 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:30:13.324429 6688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "26d2d13738244b0399f95e5e8a836d82"
format_stamp: "Formatted at 2026-09-26 11:30:13 on dist-test-slave-mrck"
server_key: "6f2e6ede0f233ce89926bb49793a0621"
server_key_iv: "4b8808bf78cae1ed32dab590f8cd54b5"
server_key_version: "encryptionkey@0"
I20260926 11:30:13.324530 6688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:13.345183 6688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:13.348368 6688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:13.348583 6688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:13.349171 6688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:13.350004 6688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:13.350076 6688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:13.350147 6688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:13.350200 6688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:13.359246 6688 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42249
I20260926 11:30:13.359263 6816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42249 every 8 connection(s)
I20260926 11:30:13.360208 6688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:30:13.364617 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 6688
I20260926 11:30:13.364758 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:30:13.365042 392 external_mini_cluster.cc:1428] Setting key 450444f4250916c2b30c916353102c0b
Sep 26 11:30:13 dist-test-slave-mrck krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422213, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 11:30:13.375103 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:13.362019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37973 (local address 127.0.98.62:33195)
0926 11:30:13.362153 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:13.362157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:13.363069 (+ 912us) server_negotiation.cc:407] Connection header received
0926 11:30:13.364493 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:13.364496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:13.364549 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:13.364630 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:13.366013 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.366642 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.367850 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.368099 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.368216 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:13.371144 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:13.371148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:13.371150 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:13.371174 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:13.372912 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:13.373428 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:13.373432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:13.373435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:13.373547 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:13.373883 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:13.373889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:13.373893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:13.374145 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:13.374261 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:13.374748 (+ 487us) server_negotiation.cc:299] Negotiation successful
0926 11:30:13.374883 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 11:30:13.375986 6819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:13.362404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.98.2:37973)
0926 11:30:13.362885 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:13.362936 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:13.364274 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:13.364849 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:13.364858 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:13.365285 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:13.365872 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:13.365889 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.366770 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.366774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:13.367674 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:13.367683 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.368254 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.368266 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:13.368434 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:13.369079 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:13.369095 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:13.370966 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:13.373040 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:13.373044 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:13.373056 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:13.373313 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:13.373642 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:13.373644 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:13.373645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:13.373783 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:13.374259 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:13.374266 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:13.374501 (+ 235us) client_negotiation.cc:769] Sending connection context
0926 11:30:13.374662 (+ 161us) client_negotiation.cc:240] Negotiation successful
0926 11:30:13.374992 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":123,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260926 11:30:13.377223 6817 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33195
I20260926 11:30:13.377491 6817 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:13.377978 6817 heartbeater.cc:511] Master 127.0.98.62:33195 requested a full tablet report, sending...
I20260926 11:30:13.379051 6492 ts_manager.cc:194] Registered new tserver with Master: 26d2d13738244b0399f95e5e8a836d82 (127.0.98.2:42249)
I20260926 11:30:13.379884 6492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 11:30:13.393004 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33195
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:13.502419 6822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:13.502665 6822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:13.502707 6822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:13.506659 6822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:13.506716 6822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:13.506770 6822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:30:13.512394 6822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:13.513327 6822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:13.514276 6822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:13.523663 6830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:13.523658 6827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:13.523878 6822 server_base.cc:1034] running on GCE node
W20260926 11:30:13.523777 6828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:13.524432 6822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:13.525043 6822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:13.526232 6822 hybrid_clock.cc:648] HybridClock initialized: now 1790422213526204 us; error 39 us; skew 500 ppm
Sep 26 11:30:13 dist-test-slave-mrck krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422213, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:13.529107 6822 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 11:30:13.529991 6822 webserver.cc:460] Webserver started at http://127.0.98.3:36463/ using document root <none> and password file <none>
I20260926 11:30:13.530599 6822 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:13.530691 6822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:13.530931 6822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:13.533074 6822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/data/instance:
uuid: "5fb677408a224fa2b4a8eee4dc918823"
format_stamp: "Formatted at 2026-09-26 11:30:13 on dist-test-slave-mrck"
server_key: "f4ed163dc956f1600b2e8c8f01e9257a"
server_key_iv: "a7822abda27a8f85348dc4e7323d9dca"
server_key_version: "encryptionkey@0"
I20260926 11:30:13.533670 6822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5fb677408a224fa2b4a8eee4dc918823"
format_stamp: "Formatted at 2026-09-26 11:30:13 on dist-test-slave-mrck"
server_key: "f4ed163dc956f1600b2e8c8f01e9257a"
server_key_iv: "a7822abda27a8f85348dc4e7323d9dca"
server_key_version: "encryptionkey@0"
I20260926 11:30:13.537873 6822 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 11:30:13.540280 6837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:13.541462 6822 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:30:13.541587 6822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "5fb677408a224fa2b4a8eee4dc918823"
format_stamp: "Formatted at 2026-09-26 11:30:13 on dist-test-slave-mrck"
server_key: "f4ed163dc956f1600b2e8c8f01e9257a"
server_key_iv: "a7822abda27a8f85348dc4e7323d9dca"
server_key_version: "encryptionkey@0"
I20260926 11:30:13.541688 6822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:13.557456 6822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:13.560304 6822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:13.560500 6822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:13.561071 6822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:13.561848 6822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:13.561914 6822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:13.561980 6822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:13.562026 6822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:13.570398 6822 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46037
I20260926 11:30:13.570430 6950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46037 every 8 connection(s)
I20260926 11:30:13.571252 6822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:30:13.579239 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 6822
I20260926 11:30:13.579361 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:30:13.579684 392 external_mini_cluster.cc:1428] Setting key dec73c17e37cdb4a2104a6a52bc30f50
Sep 26 11:30:13 dist-test-slave-mrck krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422213, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 11:30:13.584894 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:13.573023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44463 (local address 127.0.98.62:33195)
0926 11:30:13.573158 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:13.573161 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:13.573977 (+ 816us) server_negotiation.cc:407] Connection header received
0926 11:30:13.575220 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:13.575223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:13.575268 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:13.575343 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:13.576512 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.577034 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.578084 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.578288 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.578360 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:13.581339 (+ 2979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:13.581343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:13.581348 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:13.581396 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:13.583045 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:13.583515 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:13.583518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:13.583519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:13.583593 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:13.583961 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:13.583965 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:13.583967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:13.584143 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:13.584221 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:13.584688 (+ 467us) server_negotiation.cc:299] Negotiation successful
0926 11:30:13.584783 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 11:30:13.585680 6953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:13.573407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.98.3:44463)
0926 11:30:13.573831 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:13.573871 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:13.575036 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:13.575491 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:13.575497 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:13.575929 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:13.576389 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:13.576402 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.577135 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.577137 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:13.577981 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:13.577989 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.578442 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.578446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:13.578605 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:13.579158 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:13.579179 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:13.581135 (+ 1956us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:13.583144 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:13.583149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:13.583160 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:13.583425 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:13.583675 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:13.583677 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:13.583678 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:13.583875 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:13.584230 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:13.584234 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:13.584475 (+ 241us) client_negotiation.cc:769] Sending connection context
0926 11:30:13.584623 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 11:30:13.584860 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":111,"threads_started":1}
I20260926 11:30:13.586836 6951 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33195
I20260926 11:30:13.587088 6951 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:13.587577 6951 heartbeater.cc:511] Master 127.0.98.62:33195 requested a full tablet report, sending...
I20260926 11:30:13.588625 6492 ts_manager.cc:194] Registered new tserver with Master: 5fb677408a224fa2b4a8eee4dc918823 (127.0.98.3:46037)
I20260926 11:30:13.589179 6492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44463
I20260926 11:30:13.593914 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260926 11:30:13.618788 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41339
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33195,127.0.98.61:41339
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:13.727654 6956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:13.728017 6956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:13.728121 6956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:13.732256 6956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:30:13.732337 6956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:13.732373 6956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:13.732438 6956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:30:13.732476 6956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:30:13.737787 6956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33195,127.0.98.61:41339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
W20260926 11:30:13.738044 6956 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:33195,127.0.98.61:41339'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 11:30:13.738951 6956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:13.740029 6956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:13.748251 6964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:13.748252 6961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:13.748251 6962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:13.748504 6956 server_base.cc:1034] running on GCE node
I20260926 11:30:13.749096 6956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:13.750109 6956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:13.751291 6956 hybrid_clock.cc:648] HybridClock initialized: now 1790422213751257 us; error 50 us; skew 500 ppm
Sep 26 11:30:13 dist-test-slave-mrck krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422213, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:13.753952 6956 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260926 11:30:13.754781 6956 webserver.cc:460] Webserver started at http://127.0.98.61:36123/ using document root <none> and password file <none>
I20260926 11:30:13.755347 6956 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:13.755441 6956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:13.755666 6956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:13.757642 6956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-1/data/instance:
uuid: "bba2460c1d254f99900a6f71741eae0a"
format_stamp: "Formatted at 2026-09-26 11:30:13 on dist-test-slave-mrck"
server_key: "a7d9eca4f39de78706dde8f46b0cede6"
server_key_iv: "bae65741fdd3c05b50b01d32215e1ccf"
server_key_version: "encryptionkey@0"
I20260926 11:30:13.758191 6956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-1/wal/instance:
uuid: "bba2460c1d254f99900a6f71741eae0a"
format_stamp: "Formatted at 2026-09-26 11:30:13 on dist-test-slave-mrck"
server_key: "a7d9eca4f39de78706dde8f46b0cede6"
server_key_iv: "bae65741fdd3c05b50b01d32215e1ccf"
server_key_version: "encryptionkey@0"
I20260926 11:30:13.761757 6956 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 11:30:13.764065 6971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:13.765223 6956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 11:30:13.765362 6956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-1/wal
uuid: "bba2460c1d254f99900a6f71741eae0a"
format_stamp: "Formatted at 2026-09-26 11:30:13 on dist-test-slave-mrck"
server_key: "a7d9eca4f39de78706dde8f46b0cede6"
server_key_iv: "bae65741fdd3c05b50b01d32215e1ccf"
server_key_version: "encryptionkey@0"
I20260926 11:30:13.765465 6956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:13.780146 6956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:13.783242 6956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:13.783445 6956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:13.790321 6956 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41339
I20260926 11:30:13.790601 7023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41339 every 8 connection(s)
I20260926 11:30:13.791308 6956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-1/data/info.pb
I20260926 11:30:13.794076 7024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:13.794507 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 6956
I20260926 11:30:13.798404 7024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 11:30:13 dist-test-slave-mrck krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 26 11:30:13 dist-test-slave-mrck krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422213, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 11:30:13.809176 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:13.796011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59120 (local address 127.0.98.61:41339)
0926 11:30:13.796438 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:13.796453 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:13.796487 (+ 34us) server_negotiation.cc:407] Connection header received
0926 11:30:13.797138 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:13.797154 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:13.797472 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:13.797803 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:13.798575 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.799589 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.800679 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.801362 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.801522 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:13.804260 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:13.804268 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:13.804282 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:13.804304 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:13.806082 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:13.806488 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:13.806494 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:13.806500 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:13.806602 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:13.806887 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:13.806892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:13.806894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:13.807256 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:13.807460 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:13.807802 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 11:30:13.808156 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":135,"threads_started":1}
I20260926 11:30:13.809228 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:13.800056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59750 (local address 127.0.98.62:33195)
0926 11:30:13.800196 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:13.800200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:13.800532 (+ 332us) server_negotiation.cc:407] Connection header received
0926 11:30:13.800662 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:13.800667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:13.800721 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:13.800783 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:13.801559 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.802261 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.803188 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.803406 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.803485 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:13.805788 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:13.805791 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:13.805794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:13.805821 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:13.807317 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:13.807772 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:13.807776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:13.807778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:13.807884 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:13.808214 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:13.808219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:13.808220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:13.808472 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:13.808556 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:13.809001 (+ 445us) server_negotiation.cc:299] Negotiation successful
0926 11:30:13.809124 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 11:30:13.809309 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:13.799999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.0.1:59750)
0926 11:30:13.800403 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:13.800448 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:13.800565 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:13.800848 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:13.800853 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:13.801297 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:13.801455 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:13.801461 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.802389 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.802391 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:13.803067 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:13.803076 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.803546 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.803550 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:13.803671 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:13.804228 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:13.804242 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:13.805652 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:13.807415 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:13.807425 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:13.807430 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:13.807640 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:13.808005 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:13.808008 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:13.808010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:13.808100 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:13.808671 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:13.808679 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:13.808800 (+ 121us) client_negotiation.cc:769] Sending connection context
0926 11:30:13.808950 (+ 150us) client_negotiation.cc:240] Negotiation successful
0926 11:30:13.809107 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":106,"threads_started":1}
I20260926 11:30:13.811048 7024 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 11:30:13 dist-test-slave-mrck krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422213, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 11:30:13.818212 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:13.811568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59128 (local address 127.0.98.61:41339)
0926 11:30:13.811669 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:13.811673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:13.811685 (+ 12us) server_negotiation.cc:407] Connection header received
0926 11:30:13.811802 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:13.811805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:13.811838 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:13.811922 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:13.812252 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.813046 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.813745 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.813974 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.814077 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:13.815808 (+ 1731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:13.815811 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:13.815813 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:13.815834 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:13.816940 (+ 1106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:13.817286 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:13.817288 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:13.817290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:13.817361 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:13.817573 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:13.817575 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:13.817576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:13.817718 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:13.817788 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:13.817991 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 11:30:13.818073 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 11:30:13.818230 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:13.811505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41339 (local address 127.0.0.1:59128)
0926 11:30:13.811605 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:13.811619 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:13.811690 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:13.811942 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:13.811944 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:13.812086 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:13.812153 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:13.812158 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.813156 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.813159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:13.813645 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:13.813651 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.814073 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.814076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:13.814159 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:13.814520 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:13.814530 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:13.815645 (+ 1115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:13.817033 (+ 1388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:13.817036 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:13.817037 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:13.817209 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:13.817435 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:13.817437 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:13.817439 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:13.817504 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:13.817798 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:13.817800 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:13.817867 (+ 67us) client_negotiation.cc:769] Sending connection context
0926 11:30:13.817945 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 11:30:13.818031 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 11:30:13.821141 7024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a: Bootstrap starting.
I20260926 11:30:13.823415 7024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:13.824319 7024 log.cc:824] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a: Log is configured to *not* fsync() on all Append() calls
I20260926 11:30:13.826211 7024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a: No bootstrap required, opened a new log
I20260926 11:30:13.829392 7024 raft_consensus.cc:348] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } }
I20260926 11:30:13.829586 7024 raft_consensus.cc:374] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:13.829660 7024 raft_consensus.cc:729] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bba2460c1d254f99900a6f71741eae0a, State: Initialized, Role: FOLLOWER
I20260926 11:30:13.830070 7024 consensus_queue.cc:260] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } }
I20260926 11:30:13.830543 7030 sys_catalog.cc:455] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } } }
I20260926 11:30:13.830689 7030 sys_catalog.cc:458] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:30:13.831317 7024 sys_catalog.cc:564] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [sys.catalog]: configured and running, proceeding with master startup.
W20260926 11:30:13.836035 7041 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:30:13.836118 7041 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:30:13.838362 6956 master_runner.cc:428] Detected that this master bba2460c1d254f99900a6f71741eae0a is joining an existing cluster
I20260926 11:30:13.838423 6956 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:41339
I20260926 11:30:13.849370 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:13.840687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.0.1:59758)
0926 11:30:13.841079 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:13.841100 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:13.841183 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:13.841447 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:13.841450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:13.841723 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:13.841894 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:13.841902 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.842965 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.842968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:13.843871 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:13.843877 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.844361 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.844365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:13.844458 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:13.845483 (+ 1025us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:13.845494 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:13.845899 (+ 405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:13.847698 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:13.847702 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:13.847704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:13.848026 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:13.848311 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:13.848314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:13.848316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:13.848410 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:13.848828 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:13.848831 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:13.848933 (+ 102us) client_negotiation.cc:769] Sending connection context
0926 11:30:13.849040 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 11:30:13.849169 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":4,"thread_start_us":93,"threads_started":1}
I20260926 11:30:13.849388 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:13.840749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59758 (local address 127.0.98.62:33195)
0926 11:30:13.840875 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:13.840880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:13.841174 (+ 294us) server_negotiation.cc:407] Connection header received
0926 11:30:13.841253 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:13.841256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:13.841310 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:13.841399 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:13.842019 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.842859 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.843976 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.844245 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.844345 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:13.846029 (+ 1684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:13.846033 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:13.846035 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:13.846059 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:13.847568 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:13.848124 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:13.848129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:13.848131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:13.848214 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:13.848513 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:13.848520 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:13.848525 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:13.848728 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:13.848828 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:13.849115 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 11:30:13.849231 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 11:30:13.849388 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:13.840691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59140 (local address 127.0.98.61:41339)
0926 11:30:13.841027 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:13.841031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:13.841449 (+ 418us) server_negotiation.cc:407] Connection header received
0926 11:30:13.841596 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:13.841600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:13.841641 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:13.841719 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:13.842168 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.842642 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.843832 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.844207 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.844308 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:13.845870 (+ 1562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:13.845874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:13.845876 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:13.845897 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:13.847587 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:13.848192 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:13.848195 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:13.848197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:13.848270 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:13.848534 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:13.848536 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:13.848538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:13.848738 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:13.848806 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:13.849127 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 11:30:13.849231 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":263}
I20260926 11:30:13.850461 7048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:13.840775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41339 (local address 127.0.0.1:59140)
0926 11:30:13.841351 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:13.841392 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:13.841493 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:13.841748 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:13.841751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:13.841951 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:13.842069 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:13.842074 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.842770 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.842775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:13.843676 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:13.843684 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.844419 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.844423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:13.844573 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:13.845170 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:13.845185 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:13.845736 (+ 551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:13.847734 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:13.847737 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:13.847739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:13.848084 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:13.848360 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:13.848365 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:13.848366 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:13.848433 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:13.848876 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:13.848879 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:13.848953 (+ 74us) client_negotiation.cc:769] Sending connection context
0926 11:30:13.850157 (+ 1204us) client_negotiation.cc:240] Negotiation successful
0926 11:30:13.850285 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"spinlock_wait_cycles":1920,"thread_start_us":69,"threads_started":1}
Sep 26 11:30:13 dist-test-slave-mrck krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422212, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 11:30:13.864131 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:13.854290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59148 (local address 127.0.98.61:41339)
0926 11:30:13.854402 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:13.854406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:13.854730 (+ 324us) server_negotiation.cc:407] Connection header received
0926 11:30:13.854904 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:13.854907 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:13.854956 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:13.855037 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:13.855626 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.856396 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.857438 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.857772 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.857871 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:13.860427 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:13.860431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:13.860433 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:13.860456 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:13.862169 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:13.862692 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:13.862699 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:13.862703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:13.862787 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:13.863084 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:13.863090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:13.863095 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:13.863329 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:13.863431 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:13.863843 (+ 412us) server_negotiation.cc:299] Negotiation successful
0926 11:30:13.863983 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 11:30:13.864158 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:13.854236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41339 (local address 127.0.0.1:59148)
0926 11:30:13.854606 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:13.854655 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:13.854779 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:13.855072 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:13.855076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:13.855426 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:13.855524 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:13.855529 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.856493 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.856496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:13.857321 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:13.857331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.857890 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.857894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:13.857991 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:13.858513 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:13.858527 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:13.860277 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:13.862282 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:13.862290 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:13.862296 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:13.862586 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:13.862889 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:13.862892 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:13.862895 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:13.862986 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:13.863443 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:13.863450 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:13.863579 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 11:30:13.863796 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 11:30:13.863993 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":135,"threads_started":1}
I20260926 11:30:13.865869 6492 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:41339: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "bba2460c1d254f99900a6f71741eae0a"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 41339
}
attrs {
promote: true
}
}
dest_uuid: "f281a9b384774f93b3ca169c35839d66"
cas_config_opid_index: -1
I20260926 11:30:13.866516 6492 consensus_queue.cc:237] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260926 11:30:13.872932 6998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:13.872653 7052 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER bba2460c1d254f99900a6f71741eae0a (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: true } } }
I20260926 11:30:13.875042 6998 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [term 1 FOLLOWER]: Refusing update from remote peer f281a9b384774f93b3ca169c35839d66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 11:30:13.875099 7052 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:41339
I20260926 11:30:13.875417 7052 sys_catalog.cc:455] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f281a9b384774f93b3ca169c35839d66" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: true } } }
I20260926 11:30:13.875507 7052 sys_catalog.cc:458] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:13.875649 6956 master.cc:537] Master@127.0.98.61:41339 shutting down...
I20260926 11:30:13.876145 7051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
W20260926 11:30:13.878768 6479 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 11:30:13.880899 6479 consensus_peers.cc:487] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 -> Peer bba2460c1d254f99900a6f71741eae0a (127.0.98.61:41339): Couldn't send request to peer bba2460c1d254f99900a6f71741eae0a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 11:30:13.881892 6956 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:30:13.882195 6956 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:30:13.882313 6956 tablet_replica.cc:330] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a: stopping tablet replica
I20260926 11:30:13.887584 6956 master.cc:552] Master@127.0.98.61:41339 shutdown complete.
I20260926 11:30:13.887640 6956 master_runner.cc:305] Clearing existing system tablet
I20260926 11:30:13.887974 6956 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:13.888036 6956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:13.889253 6956 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 11:30:13.890655 7057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:13.890930 6956 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:13.891062 6956 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 11:30:13.891106 6956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-1/wal
uuid: "bba2460c1d254f99900a6f71741eae0a"
format_stamp: "Formatted at 2026-09-26 11:30:13 on dist-test-slave-mrck"
server_key: "a7d9eca4f39de78706dde8f46b0cede6"
server_key_iv: "bae65741fdd3c05b50b01d32215e1ccf"
server_key_version: "encryptionkey@0"
I20260926 11:30:13.892911 6956 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:30:13.895653 6956 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 11:30:13.895772 6956 log.cc:1196] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 11:30:13.896023 6956 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a: Deleting consensus metadata
I20260926 11:30:13.896209 6956 master_runner.cc:315] Copying system tablet from 127.0.98.62:33195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260926 11:30:13.897317 6956 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:33195
I20260926 11:30:13.898316 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39305
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33195,127.0.98.61:41339,127.0.98.60:39305
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:30:13.908349 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:13.898024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59760 (local address 127.0.98.62:33195)
0926 11:30:13.898177 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:13.898184 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:13.899663 (+ 1479us) server_negotiation.cc:407] Connection header received
0926 11:30:13.899871 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:13.899874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:13.899927 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:13.900016 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:13.900456 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.900982 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.902313 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.902664 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.902782 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:13.904480 (+ 1698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:13.904485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:13.904488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:13.904516 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:13.906292 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:13.906820 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:13.906823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:13.906825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:13.906910 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:13.907253 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:13.907256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:13.907258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:13.907465 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:13.907559 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:13.907990 (+ 431us) server_negotiation.cc:299] Negotiation successful
0926 11:30:13.908111 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 11:30:13.908418 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:13.897896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.0.1:59760)
0926 11:30:13.899551 (+ 1655us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:13.899565 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:13.899714 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:13.900032 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:13.900035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:13.900232 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:13.900331 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:13.900337 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.901094 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.901101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:13.902170 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:13.902184 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:13.902802 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:13.902808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:13.902940 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:13.903546 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:13.903568 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:13.904122 (+ 554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:13.906412 (+ 2290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:13.906418 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:13.906424 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:13.906709 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:13.907010 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:13.907015 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:13.907018 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:13.907122 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:13.907567 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:13.907572 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:13.907700 (+ 128us) client_negotiation.cc:769] Sending connection context
0926 11:30:13.907917 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 11:30:13.908130 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1589,"thread_start_us":87,"threads_started":1}
I20260926 11:30:13.909348 6512 tablet_copy_service.cc:140] P f281a9b384774f93b3ca169c35839d66: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer bba2460c1d254f99900a6f71741eae0a ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:59760)
I20260926 11:30:13.909525 6512 tablet_copy_service.cc:161] P f281a9b384774f93b3ca169c35839d66: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer bba2460c1d254f99900a6f71741eae0a at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:59760: session id = bba2460c1d254f99900a6f71741eae0a-00000000000000000000000000000000
I20260926 11:30:13.911280 6512 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 11:30:13.913123 6956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 11:30:13.915584 6956 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a: tablet copy: Starting download of 0 data blocks...
I20260926 11:30:13.915822 6956 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a: tablet copy: Starting download of 1 WAL segments...
I20260926 11:30:13.917544 6956 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 11:30:13.919360 6512 tablet_copy_service.cc:342] P f281a9b384774f93b3ca169c35839d66: Request end of tablet copy session bba2460c1d254f99900a6f71741eae0a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:59760
I20260926 11:30:13.919480 6512 tablet_copy_service.cc:433] P f281a9b384774f93b3ca169c35839d66: ending tablet copy session bba2460c1d254f99900a6f71741eae0a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer bba2460c1d254f99900a6f71741eae0a
I20260926 11:30:13.927047 6956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:13.928900 7065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:13.933089 6956 server_base.cc:1034] running on GCE node
W20260926 11:30:13.933262 7066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:13.933370 7068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:13.933575 6956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:13.933898 6956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:13.935037 6956 hybrid_clock.cc:648] HybridClock initialized: now 1790422213935015 us; error 41 us; skew 500 ppm
Sep 26 11:30:13 dist-test-slave-mrck krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422213, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:13.936671 6956 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260926 11:30:13.937130 6956 webserver.cc:460] Webserver started at http://127.0.98.61:41091/ using document root <none> and password file <none>
I20260926 11:30:13.937316 6956 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:13.937357 6956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:13.938681 6956 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 11:30:13.939698 7075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:13.940035 6956 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 11:30:13.940122 6956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-1/wal
uuid: "bba2460c1d254f99900a6f71741eae0a"
format_stamp: "Formatted at 2026-09-26 11:30:13 on dist-test-slave-mrck"
server_key: "a7d9eca4f39de78706dde8f46b0cede6"
server_key_iv: "bae65741fdd3c05b50b01d32215e1ccf"
server_key_version: "encryptionkey@0"
I20260926 11:30:13.940202 6956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:13.949815 6956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:13.951897 6956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:13.956365 6956 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41339
I20260926 11:30:13.956957 6956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-1/data/info.pb
I20260926 11:30:13.958765 7128 sys_catalog.cc:263] Verifying existing consensus state
I20260926 11:30:13.959833 7128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a: Bootstrap starting.
I20260926 11:30:13.959909 7127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41339 every 8 connection(s)
I20260926 11:30:13.970021 7128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 11:30:13.970333 7128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a: Bootstrap complete.
I20260926 11:30:13.970913 7128 raft_consensus.cc:348] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: true } }
I20260926 11:30:13.971048 7128 raft_consensus.cc:729] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bba2460c1d254f99900a6f71741eae0a, State: Initialized, Role: LEARNER
I20260926 11:30:13.971159 7128 consensus_queue.cc:260] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: true } }
I20260926 11:30:13.971457 7131 sys_catalog.cc:455] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: true } } }
I20260926 11:30:13.971594 7128 sys_catalog.cc:564] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:30:13.971635 7131 sys_catalog.cc:458] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [sys.catalog]: This master's current role is: LEARNER
I20260926 11:30:13.975121 7142 catalog_manager.cc:1259] Loaded cluster ID: 7d38a3fb7f064528b10e3eedc9ccf1ce
I20260926 11:30:13.975176 7142 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a: loading cluster ID for follower catalog manager: success
I20260926 11:30:13.977221 7142 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a: acquiring CA information for follower catalog manager: success
I20260926 11:30:13.978531 7142 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 11:30:14.025552 7064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:14.025835 7064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:14.025918 7064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:14.030103 7064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:30:14.030174 7064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:14.030200 7064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:14.030257 7064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:30:14.030293 7064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:30:14.035609 7064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33195,127.0.98.61:41339,127.0.98.60:39305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:14.036743 7064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:14.037799 7064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:14.046452 7147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:14.046489 7150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:14.046460 7148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:14.046868 7064 server_base.cc:1034] running on GCE node
I20260926 11:30:14.047420 7064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:14.048569 7064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:14.049757 7064 hybrid_clock.cc:648] HybridClock initialized: now 1790422214049743 us; error 27 us; skew 500 ppm
Sep 26 11:30:14 dist-test-slave-mrck krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422214, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:14.052641 7064 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260926 11:30:14.053478 7064 webserver.cc:460] Webserver started at http://127.0.98.60:46281/ using document root <none> and password file <none>
I20260926 11:30:14.054015 7064 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:14.054105 7064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:14.054327 7064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:14.056393 7064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-2/data/instance:
uuid: "b5d89fa8510448b9950d2ef424068e4d"
format_stamp: "Formatted at 2026-09-26 11:30:14 on dist-test-slave-mrck"
server_key: "702bdc342f2105db80e43437f3e5e2b4"
server_key_iv: "0119260c5d0c5331c64f93340e091fa5"
server_key_version: "encryptionkey@0"
I20260926 11:30:14.056947 7064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-2/wal/instance:
uuid: "b5d89fa8510448b9950d2ef424068e4d"
format_stamp: "Formatted at 2026-09-26 11:30:14 on dist-test-slave-mrck"
server_key: "702bdc342f2105db80e43437f3e5e2b4"
server_key_iv: "0119260c5d0c5331c64f93340e091fa5"
server_key_version: "encryptionkey@0"
I20260926 11:30:14.060487 7064 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:30:14.062834 7157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:14.064153 7064 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:30:14.064289 7064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-2/wal
uuid: "b5d89fa8510448b9950d2ef424068e4d"
format_stamp: "Formatted at 2026-09-26 11:30:14 on dist-test-slave-mrck"
server_key: "702bdc342f2105db80e43437f3e5e2b4"
server_key_iv: "0119260c5d0c5331c64f93340e091fa5"
server_key_version: "encryptionkey@0"
I20260926 11:30:14.064409 7064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:14.086237 7064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:14.089452 7064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:14.089668 7064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:14.098158 7064 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39305
I20260926 11:30:14.098384 7209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39305 every 8 connection(s)
I20260926 11:30:14.099248 7064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-2/data/info.pb
I20260926 11:30:14.101989 7210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:14.105722 7210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:30:14.106352 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 7064
Sep 26 11:30:14 dist-test-slave-mrck krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422214, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 26 11:30:14 dist-test-slave-mrck krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 11:30:14.119715 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.107854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59770 (local address 127.0.98.62:33195)
0926 11:30:14.108020 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:14.108027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:14.108574 (+ 547us) server_negotiation.cc:407] Connection header received
0926 11:30:14.109564 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:14.109571 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:14.109632 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:14.109708 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:14.110803 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.111338 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.112850 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.113115 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.113214 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.115935 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:14.115939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:14.115942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:14.115977 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:14.117662 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.118165 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.118168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.118170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.118253 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.118628 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.118631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.118633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.118817 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:14.118921 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:14.119429 (+ 508us) server_negotiation.cc:299] Negotiation successful
0926 11:30:14.119546 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 11:30:14.120584 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.107820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.0.1:59770)
0926 11:30:14.108448 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:14.108485 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:14.109220 (+ 735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:14.109775 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:14.109783 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:14.110274 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:14.110661 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.110682 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.111469 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.111473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.112727 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.112737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.113263 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.113266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.113405 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.113995 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:14.114015 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:14.115773 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:14.117788 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.117798 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.117808 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.118056 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.118362 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.118365 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.118368 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.118521 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.118929 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:14.118938 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:14.119209 (+ 271us) client_negotiation.cc:769] Sending connection context
0926 11:30:14.119372 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 11:30:14.119675 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":31,"spinlock_wait_cycles":55680,"thread_start_us":187,"threads_started":1}
I20260926 11:30:14.120586 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.107819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46570 (local address 127.0.98.60:39305)
0926 11:30:14.108315 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:14.108342 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:14.108381 (+ 39us) server_negotiation.cc:407] Connection header received
0926 11:30:14.109182 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:14.109200 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:14.109503 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:14.109692 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:14.110657 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.111796 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.113050 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.113510 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.113588 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.116378 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:14.116387 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:14.116403 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:14.116422 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:14.117938 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.118375 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.118387 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.118388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.118535 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.118820 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.118840 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.118842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.119199 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:14.119392 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:14.119624 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 11:30:14.119782 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":320,"thread_start_us":146,"threads_started":1}
I20260926 11:30:14.121832 7210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 11:30:14 dist-test-slave-mrck krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422214, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 11:30:14.129765 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.122297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41339 (local address 127.0.0.1:59154)
0926 11:30:14.122412 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:14.122429 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:14.122497 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:14.122986 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:14.122988 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:14.123113 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:14.123194 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.123198 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.124136 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.124139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.124648 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.124655 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.125031 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.125034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.125100 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.125553 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:14.125564 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:14.126685 (+ 1121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:14.128467 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.128474 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.128476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.128653 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.128956 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.128958 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.128959 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.129028 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.129390 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:14.129393 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:14.129472 (+ 79us) client_negotiation.cc:769] Sending connection context
0926 11:30:14.129564 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 11:30:14.129661 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 11:30:14.129880 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.122498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59154 (local address 127.0.98.61:41339)
0926 11:30:14.122756 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:14.122760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:14.122771 (+ 11us) server_negotiation.cc:407] Connection header received
0926 11:30:14.122804 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:14.122806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:14.122844 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:14.122950 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:14.123296 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.124049 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.124738 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.124946 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.125018 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.126804 (+ 1786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:14.126807 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:14.126809 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:14.126830 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:14.128333 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.128783 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.128787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.128792 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.128878 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.129104 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.129106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.129107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.129287 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:14.129361 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:14.129617 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 11:30:14.129710 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":112,"threads_started":1}
I20260926 11:30:14.130582 7210 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39305 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 11:30:14 dist-test-slave-mrck krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422214, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 11:30:14.137740 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.131034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39305 (local address 127.0.0.1:46580)
0926 11:30:14.131128 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:14.131139 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:14.131193 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:14.131395 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:14.131397 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:14.131513 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:14.131576 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.131580 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.132391 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.132394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.132899 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.132907 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.133366 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.133369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.133429 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.133897 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:14.133908 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:14.135069 (+ 1161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:14.136589 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.136591 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.136592 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.136768 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.136997 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.136999 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.137000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.137067 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.137366 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:14.137368 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:14.137438 (+ 70us) client_negotiation.cc:769] Sending connection context
0926 11:30:14.137517 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 11:30:14.137604 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 11:30:14.137766 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.131126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46580 (local address 127.0.98.60:39305)
0926 11:30:14.131240 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:14.131243 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:14.131253 (+ 10us) server_negotiation.cc:407] Connection header received
0926 11:30:14.131280 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:14.131282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:14.131312 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:14.131373 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:14.131662 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.132273 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.133016 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.133241 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.133321 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.135183 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:14.135186 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:14.135188 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:14.135206 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:14.136497 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.136852 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.136854 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.136855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.136918 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.137137 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.137138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.137139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.137277 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:14.137403 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:14.137568 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 11:30:14.137647 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 11:30:14.140897 7210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d: Bootstrap starting.
I20260926 11:30:14.143205 7210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:14.144140 7210 log.cc:824] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d: Log is configured to *not* fsync() on all Append() calls
I20260926 11:30:14.148633 7210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d: No bootstrap required, opened a new log
I20260926 11:30:14.153046 7210 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39305 } }
I20260926 11:30:14.153258 7210 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:14.153291 7210 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5d89fa8510448b9950d2ef424068e4d, State: Initialized, Role: FOLLOWER
I20260926 11:30:14.153672 7210 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39305 } }
I20260926 11:30:14.154068 7216 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39305 } } }
I20260926 11:30:14.154208 7216 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:30:14.154834 7210 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [sys.catalog]: configured and running, proceeding with master startup.
W20260926 11:30:14.159456 7227 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:30:14.159533 7227 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:30:14.162918 7064 master_runner.cc:428] Detected that this master b5d89fa8510448b9950d2ef424068e4d is joining an existing cluster
I20260926 11:30:14.162981 7064 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:39305
I20260926 11:30:14.165372 6683 heartbeater.cc:503] Master 127.0.98.62:33195 was elected leader, sending a full tablet report...
I20260926 11:30:14.174273 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.165355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59156 (local address 127.0.98.61:41339)
0926 11:30:14.165665 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:14.165671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:14.166160 (+ 489us) server_negotiation.cc:407] Connection header received
0926 11:30:14.166333 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:14.166336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:14.166385 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:14.166435 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:14.167112 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.167633 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.168668 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.169026 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.169141 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.170510 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:14.170514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:14.170516 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:14.170542 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:14.172496 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.173013 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.173017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.173019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.173101 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.173388 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.173391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.173393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.173585 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:14.173652 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:14.173976 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 11:30:14.174104 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232}
I20260926 11:30:14.174377 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 6554
I20260926 11:30:14.174857 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.165462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41339 (local address 127.0.0.1:59156)
0926 11:30:14.166080 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:14.166094 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:14.166229 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:14.166580 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:14.166582 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:14.166835 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:14.167005 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.167012 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.167827 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.167830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.168546 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.168554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.169269 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.169272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.169358 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.169928 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:14.169943 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:14.170387 (+ 444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:14.172617 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.172620 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.172622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.172903 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.173208 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.173210 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.173212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.173291 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.173729 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:14.173732 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:14.173810 (+ 78us) client_negotiation.cc:769] Sending connection context
0926 11:30:14.174474 (+ 664us) client_negotiation.cc:240] Negotiation successful
0926 11:30:14.174596 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":72,"threads_started":1}
I20260926 11:30:14.180064 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.165960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46584 (local address 127.0.98.60:39305)
0926 11:30:14.166065 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:14.166071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:14.172115 (+ 6044us) server_negotiation.cc:407] Connection header received
0926 11:30:14.172205 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:14.172208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:14.172265 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:14.172623 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:14.173184 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.173915 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.175777 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.176035 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.176090 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.177114 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:14.177117 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:14.177119 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:14.177140 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:14.178519 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.178913 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.178919 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.178922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.178989 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.179256 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.179261 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.179263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.179438 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:14.179543 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:14.179805 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 11:30:14.179942 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":30}
I20260926 11:30:14.180075 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.165769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39305 (local address 127.0.0.1:46584)
0926 11:30:14.172006 (+ 6237us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:14.172021 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:14.172129 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:14.172733 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:14.172736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:14.172951 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:14.173078 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.173085 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.174037 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.174039 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.175650 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.175658 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.176182 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.176184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.176247 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.176684 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:14.176704 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:14.176984 (+ 280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:14.178625 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.178631 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.178633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.178822 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.179079 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.179082 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.179084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.179159 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.179532 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:14.179534 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:14.179628 (+ 94us) client_negotiation.cc:769] Sending connection context
0926 11:30:14.179733 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 11:30:14.179887 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6164,"thread_start_us":64,"threads_started":1}
I20260926 11:30:14.183030 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46203
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33195
--webserver_port=45105
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:33195,127.0.98.61:41339
--tserver_master_addrs=127.0.98.62:33195,127.0.98.61:41339,127.0.98.60:39305 with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:30:14.184324 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.165356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59772 (local address 127.0.98.62:33195)
0926 11:30:14.165476 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:14.165481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:14.165923 (+ 442us) server_negotiation.cc:407] Connection header received
0926 11:30:14.166104 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:14.166107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:14.166155 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:14.166207 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:14.167272 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.168120 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.169682 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.170040 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.170150 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.171463 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:14.171470 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:14.171475 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:14.171504 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:14.173255 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.183059 (+ 9804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.183063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.183065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.183158 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.183440 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.183443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.183445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.183639 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:14.183706 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:14.184058 (+ 352us) server_negotiation.cc:299] Negotiation successful
0926 11:30:14.184175 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 11:30:14.185004 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.165337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.0.1:59772)
0926 11:30:14.165843 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:14.165858 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:14.165991 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:14.166344 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:14.166347 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:14.167016 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:14.167134 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.167142 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.168717 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.168720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.169562 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.169570 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.170164 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.170168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.170278 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.170894 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:14.170908 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:14.171311 (+ 403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:14.182681 (+ 11370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.182685 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.182687 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.182934 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.183267 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.183270 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.183272 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.183346 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.183798 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:14.183801 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:14.183891 (+ 90us) client_negotiation.cc:769] Sending connection context
0926 11:30:14.184509 (+ 618us) client_negotiation.cc:240] Negotiation successful
0926 11:30:14.184839 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":98,"threads_started":1}
Sep 26 11:30:14 dist-test-slave-mrck krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422212, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 11:30:14.211647 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.189159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39305 (local address 127.0.0.1:46594)
0926 11:30:14.189315 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:14.189329 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:14.189395 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:14.189692 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:14.189695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:14.189887 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:14.189982 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.189989 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.191058 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.191061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.191952 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.191962 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.192601 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.192605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.192702 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.193212 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:14.193233 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:14.194843 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:14.209842 (+ 14999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.209846 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.209849 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.210172 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.210499 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.210502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.210504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.210609 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.211054 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:14.211057 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:14.211182 (+ 125us) client_negotiation.cc:769] Sending connection context
0926 11:30:14.211304 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 11:30:14.211473 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260926 11:30:14.211648 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.189233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46594 (local address 127.0.98.60:39305)
0926 11:30:14.189355 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:14.189361 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:14.189378 (+ 17us) server_negotiation.cc:407] Connection header received
0926 11:30:14.189505 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:14.189509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:14.189577 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:14.189669 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:14.190100 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.190941 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.192072 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.192475 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.192596 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.194980 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:14.194983 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:14.194986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:14.195010 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:14.209661 (+ 14651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.210301 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.210304 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.210306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.210398 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.210709 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.210712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.210714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.210950 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:14.211045 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:14.211366 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 11:30:14.211487 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 11:30:14.213195 6492 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:39305: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 39305
}
attrs {
promote: true
}
}
dest_uuid: "f281a9b384774f93b3ca169c35839d66"
cas_config_opid_index: 5
I20260926 11:30:14.213788 6492 consensus_queue.cc:237] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: true } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: true } }
I20260926 11:30:14.214835 7051 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b5d89fa8510448b9950d2ef424068e4d (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: true } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: true } } }
I20260926 11:30:14.216331 7051 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:39305
I20260926 11:30:14.216497 7051 sys_catalog.cc:455] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f281a9b384774f93b3ca169c35839d66" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: true } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: true } } }
I20260926 11:30:14.216594 7051 sys_catalog.cc:458] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:14.217094 7184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:14.218628 7184 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [term 1 FOLLOWER]: Refusing update from remote peer f281a9b384774f93b3ca169c35839d66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 11:30:14.219384 7064 master.cc:537] Master@127.0.98.60:39305 shutting down...
I20260926 11:30:14.219393 7052 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 11:30:14.221184 6476 consensus_peers.cc:487] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 -> Peer b5d89fa8510448b9950d2ef424068e4d (127.0.98.60:39305): Couldn't send request to peer b5d89fa8510448b9950d2ef424068e4d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 11:30:14.224500 7064 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:30:14.224762 7064 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:30:14.224862 7064 tablet_replica.cc:330] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d: stopping tablet replica
I20260926 11:30:14.225541 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.214987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41339 (local address 127.0.0.1:59168)
0926 11:30:14.215312 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:14.215326 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:14.215401 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:14.221491 (+ 6090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:14.221494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:14.221517 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:30:14.221614 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.221620 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.222684 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.222687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.223937 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.223948 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.225134 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.225137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.225231 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.225275 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 11:30:14.225342 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 11:30:14.225416 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251}
I20260926 11:30:14.225970 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.215883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59168 (local address 127.0.98.61:41339)
0926 11:30:14.221200 (+ 5317us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:14.221205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:14.221222 (+ 17us) server_negotiation.cc:407] Connection header received
0926 11:30:14.221278 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:14.221283 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:14.221347 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:14.221456 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:30:14.221748 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.222542 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.224083 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.224720 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.224797 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.224946 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:14.225759 (+ 813us) server_negotiation.cc:299] Negotiation successful
0926 11:30:14.225857 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5216}
I20260926 11:30:14.226683 7094 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [term 1 LEARNER]: Refusing update from remote peer f281a9b384774f93b3ca169c35839d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 11:30:14.227334 7052 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 11:30:14.230154 7094 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b5d89fa8510448b9950d2ef424068e4d (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: true } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: true } } }
I20260926 11:30:14.231299 7064 master.cc:552] Master@127.0.98.60:39305 shutdown complete.
I20260926 11:30:14.231381 7064 master_runner.cc:305] Clearing existing system tablet
I20260926 11:30:14.231689 7064 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:14.231807 7064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:14.232566 7131 sys_catalog.cc:455] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f281a9b384774f93b3ca169c35839d66. Latest consensus state: current_term: 1 leader_uuid: "f281a9b384774f93b3ca169c35839d66" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: true } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: true } } }
I20260926 11:30:14.232692 7131 sys_catalog.cc:458] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [sys.catalog]: This master's current role is: LEARNER
I20260926 11:30:14.232782 7131 sys_catalog.cc:455] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f281a9b384774f93b3ca169c35839d66" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: true } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: true } } }
I20260926 11:30:14.232843 7131 sys_catalog.cc:458] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [sys.catalog]: This master's current role is: LEARNER
I20260926 11:30:14.233340 7064 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 11:30:14.234119 7052 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66: attempting to promote NON_VOTER bba2460c1d254f99900a6f71741eae0a to VOTER
I20260926 11:30:14.234148 7240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:14.234401 7064 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:14.234503 7064 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 11:30:14.234532 7064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-2/wal
uuid: "b5d89fa8510448b9950d2ef424068e4d"
format_stamp: "Formatted at 2026-09-26 11:30:14 on dist-test-slave-mrck"
server_key: "702bdc342f2105db80e43437f3e5e2b4"
server_key_iv: "0119260c5d0c5331c64f93340e091fa5"
server_key_version: "encryptionkey@0"
I20260926 11:30:14.234591 7052 consensus_queue.cc:237] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: true } }
I20260926 11:30:14.236251 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.235864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39305 (local address 127.0.0.1:46596)
0926 11:30:14.236072 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:14.236156 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260926 11:30:14.236435 7064 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:30:14.236835 7094 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [term 1 LEARNER]: Refusing update from remote peer f281a9b384774f93b3ca169c35839d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 11:30:14.237296 7051 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 11:30:14.238432 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.238175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39305 (local address 127.0.0.1:46612)
0926 11:30:14.238322 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:14.238369 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260926 11:30:14.239008 6476 consensus_peers.cc:487] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 -> Peer b5d89fa8510448b9950d2ef424068e4d (127.0.98.60:39305): Couldn't send request to peer b5d89fa8510448b9950d2ef424068e4d. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39305: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 11:30:14.239238 7064 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 11:30:14.239352 7064 log.cc:1196] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 11:30:14.239647 7064 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d: Deleting consensus metadata
I20260926 11:30:14.239886 7064 master_runner.cc:315] Copying system tablet from 127.0.98.62:33195
I20260926 11:30:14.239897 7051 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bba2460c1d254f99900a6f71741eae0a (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: true } } }
I20260926 11:30:14.241056 7064 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:33195
I20260926 11:30:14.241266 7051 sys_catalog.cc:455] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f281a9b384774f93b3ca169c35839d66" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: true } } }
I20260926 11:30:14.241376 7051 sys_catalog.cc:458] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:14.241492 7051 sys_catalog.cc:455] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f281a9b384774f93b3ca169c35839d66" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: true } } }
I20260926 11:30:14.241564 7051 sys_catalog.cc:458] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:14.242904 7094 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bba2460c1d254f99900a6f71741eae0a (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: true } } }
I20260926 11:30:14.244297 7238 sys_catalog.cc:455] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f281a9b384774f93b3ca169c35839d66" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: true } } }
I20260926 11:30:14.244415 7238 sys_catalog.cc:458] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:30:14.250946 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.242031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59774 (local address 127.0.98.62:33195)
0926 11:30:14.242130 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:14.242134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:14.242291 (+ 157us) server_negotiation.cc:407] Connection header received
0926 11:30:14.242464 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:14.242467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:14.242511 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:14.242560 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:14.243220 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.244105 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.245614 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.245958 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.246028 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.247449 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:14.247453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:14.247456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:14.247482 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:14.249227 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.249693 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.249697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.249699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.249801 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.250101 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.250105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.250106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.250317 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:14.250414 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:14.250678 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 11:30:14.250790 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 11:30:14.250962 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.241960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.0.1:59774)
0926 11:30:14.242215 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:14.242229 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:14.242361 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:14.242695 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:14.242698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:14.242923 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:14.243035 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.243042 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.244563 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.244566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.245476 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.245487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.246182 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.246185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.246276 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.246855 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:14.246870 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:14.247300 (+ 430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:14.249340 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.249344 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.249346 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.249592 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.249914 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.249918 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.249920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.250008 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.250414 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:14.250418 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:14.250509 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 11:30:14.250617 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 11:30:14.250756 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":56,"threads_started":1}
I20260926 11:30:14.251560 6512 tablet_copy_service.cc:140] P f281a9b384774f93b3ca169c35839d66: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b5d89fa8510448b9950d2ef424068e4d ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:59774)
I20260926 11:30:14.251647 6512 tablet_copy_service.cc:161] P f281a9b384774f93b3ca169c35839d66: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b5d89fa8510448b9950d2ef424068e4d at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:59774: session id = b5d89fa8510448b9950d2ef424068e4d-00000000000000000000000000000000
I20260926 11:30:14.252687 6512 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 11:30:14.253985 7064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 11:30:14.256541 7064 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d: tablet copy: Starting download of 0 data blocks...
I20260926 11:30:14.256716 7064 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d: tablet copy: Starting download of 1 WAL segments...
I20260926 11:30:14.258194 7064 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 11:30:14.259951 6512 tablet_copy_service.cc:342] P f281a9b384774f93b3ca169c35839d66: Request end of tablet copy session b5d89fa8510448b9950d2ef424068e4d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:59774
I20260926 11:30:14.260027 6512 tablet_copy_service.cc:433] P f281a9b384774f93b3ca169c35839d66: ending tablet copy session b5d89fa8510448b9950d2ef424068e4d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b5d89fa8510448b9950d2ef424068e4d
I20260926 11:30:14.267473 7064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:14.269337 7248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:14.269500 7249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:14.269886 7251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:14.270047 7064 server_base.cc:1034] running on GCE node
I20260926 11:30:14.270205 7064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:14.270555 7064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:14.271780 7064 hybrid_clock.cc:648] HybridClock initialized: now 1790422214271767 us; error 26 us; skew 500 ppm
Sep 26 11:30:14 dist-test-slave-mrck krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422214, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:14.273542 7064 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260926 11:30:14.274066 7064 webserver.cc:460] Webserver started at http://127.0.98.60:46291/ using document root <none> and password file <none>
I20260926 11:30:14.274302 7064 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:14.274386 7064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:14.275866 7064 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 11:30:14.276917 7258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:14.277151 7064 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 11:30:14.277258 7064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-2/wal
uuid: "b5d89fa8510448b9950d2ef424068e4d"
format_stamp: "Formatted at 2026-09-26 11:30:14 on dist-test-slave-mrck"
server_key: "702bdc342f2105db80e43437f3e5e2b4"
server_key_iv: "0119260c5d0c5331c64f93340e091fa5"
server_key_version: "encryptionkey@0"
I20260926 11:30:14.277398 7064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:14.300678 7064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:14.303189 7064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:14.308923 7064 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39305
I20260926 11:30:14.308924 7310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39305 every 8 connection(s)
I20260926 11:30:14.309825 7064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/master-2/data/info.pb
I20260926 11:30:14.311678 7311 sys_catalog.cc:263] Verifying existing consensus state
I20260926 11:30:14.312682 7311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d: Bootstrap starting.
I20260926 11:30:14.323855 7311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 11:30:14.324167 7311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d: Bootstrap complete.
I20260926 11:30:14.324697 7311 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: true } }
I20260926 11:30:14.324851 7311 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b5d89fa8510448b9950d2ef424068e4d, State: Initialized, Role: LEARNER
I20260926 11:30:14.324955 7311 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: true } }
I20260926 11:30:14.325279 7317 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: true } } }
I20260926 11:30:14.325421 7311 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:30:14.325461 7317 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [sys.catalog]: This master's current role is: LEARNER
W20260926 11:30:14.328233 7236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:14.328773 7236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:14.328864 7236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 11:30:14.329960 7328 catalog_manager.cc:1259] Loaded cluster ID: 7d38a3fb7f064528b10e3eedc9ccf1ce
I20260926 11:30:14.330018 7328 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d: loading cluster ID for follower catalog manager: success
I20260926 11:30:14.332144 7328 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d: acquiring CA information for follower catalog manager: success
I20260926 11:30:14.333527 7328 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 11:30:14.334512 7236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:14.334564 7236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:14.334676 7236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:30:14.340299 7236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46203
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45105
--tserver_master_addrs=127.0.98.62:33195,127.0.98.61:41339,127.0.98.60:39305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:14.341444 7236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:14.342459 7236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:14.352041 7331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:14.352109 7333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:14.352041 7330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:14.352633 7236 server_base.cc:1034] running on GCE node
I20260926 11:30:14.353001 7236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:14.353574 7236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:14.354750 7236 hybrid_clock.cc:648] HybridClock initialized: now 1790422214354719 us; error 44 us; skew 500 ppm
Sep 26 11:30:14 dist-test-slave-mrck krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422214, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:14.357546 7236 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 11:30:14.358378 7236 webserver.cc:460] Webserver started at http://127.0.98.1:45105/ using document root <none> and password file <none>
I20260926 11:30:14.358944 7236 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:14.359045 7236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:14.362867 7236 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:30:14.364871 7340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:14.366031 7236 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 11:30:14.366154 7236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "4b3cc5c906eb42cca07bc2854fc9ff3c"
format_stamp: "Formatted at 2026-09-26 11:30:13 on dist-test-slave-mrck"
server_key: "ce329de21b58ba6969f98b1b4b6ca8e0"
server_key_iv: "5540601aaefd903bc46b61df1643990f"
server_key_version: "encryptionkey@0"
I20260926 11:30:14.366672 7236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:14.381901 6817 heartbeater.cc:503] Master 127.0.98.62:33195 was elected leader, sending a full tablet report...
I20260926 11:30:14.386380 7236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:14.389628 7236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:14.389817 7236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:14.390460 7236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:14.391273 7236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:14.391350 7236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:14.391440 7236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:14.391489 7236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:14.400974 7236 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46203
I20260926 11:30:14.400995 7453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46203 every 8 connection(s)
I20260926 11:30:14.401927 7236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:30:14.410470 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 7236
I20260926 11:30:14.410660 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 6688
Sep 26 11:30:14 dist-test-slave-mrck krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422214, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 26 11:30:14 dist-test-slave-mrck krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422214, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 11:30:14.418032 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.403817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49375 (local address 127.0.98.61:41339)
0926 11:30:14.403975 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:14.403980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:14.406248 (+ 2268us) server_negotiation.cc:407] Connection header received
0926 11:30:14.406450 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:14.406456 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:14.406513 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:14.406563 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:14.408390 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.409267 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.410644 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.410999 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.411103 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.414013 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:14.414016 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:14.414019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:14.414052 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:14.415705 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.416263 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.416270 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.416275 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.416375 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.416717 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.416721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.416723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.416912 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:14.417018 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:14.417630 (+ 612us) server_negotiation.cc:299] Negotiation successful
0926 11:30:14.417771 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 11:30:14.419042 7461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.405873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41339 (local address 127.0.98.1:49375)
0926 11:30:14.406156 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:14.406172 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:14.406323 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:14.406699 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:14.406701 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:14.407493 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:14.408255 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.408266 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.409408 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.409411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.410516 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.410525 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.411541 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.411544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.411673 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.412357 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:14.412376 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:14.413848 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:14.415858 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.415864 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.415880 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.416144 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.416472 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.416474 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.416476 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.416607 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.417016 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:14.417023 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:14.417334 (+ 311us) client_negotiation.cc:769] Sending connection context
0926 11:30:14.417544 (+ 210us) client_negotiation.cc:240] Negotiation successful
0926 11:30:14.417972 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":176,"thread_start_us":152,"threads_started":1}
I20260926 11:30:14.419523 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42249
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33195
--webserver_port=44247
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:33195,127.0.98.61:41339
--tserver_master_addrs=127.0.98.62:33195,127.0.98.61:41339,127.0.98.60:39305 with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 11:30:14 dist-test-slave-mrck krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422214, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 11:30:14.422068 7456 heartbeater.cc:347] Connected to a master server at 127.0.98.61:41339
I20260926 11:30:14.422308 7456 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:14.422807 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.404260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39305 (local address 127.0.98.1:54403)
0926 11:30:14.405129 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:14.405168 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:14.406144 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:14.407240 (+ 1096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:14.407245 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:14.407524 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:14.408248 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.408267 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.409428 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.409430 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.415067 (+ 5637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.415080 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.415817 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.415854 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.415986 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.416545 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:14.416565 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:14.418641 (+ 2076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:14.421018 (+ 2377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.421023 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.421027 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.421283 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.421602 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.421607 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.421611 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.421696 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.422266 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:14.422272 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:14.422418 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 11:30:14.422517 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 11:30:14.422658 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":695,"mutex_wait_us":243,"spinlock_wait_cycles":15744,"thread_start_us":558,"threads_started":1}
I20260926 11:30:14.423508 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.403765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54403 (local address 127.0.98.60:39305)
0926 11:30:14.404119 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:14.404124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:14.405302 (+ 1178us) server_negotiation.cc:407] Connection header received
0926 11:30:14.406628 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:14.406634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:14.406690 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:14.407209 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:14.408463 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.409279 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.415204 (+ 5925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.415642 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.415826 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.418803 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:14.418807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:14.418812 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:14.418847 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:14.420888 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.421397 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.421403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.421407 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.421492 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.421806 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.421812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.421814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.422064 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:14.422216 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:14.423164 (+ 948us) server_negotiation.cc:299] Negotiation successful
0926 11:30:14.423322 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":75,"threads_started":1}
I20260926 11:30:14.422813 7456 heartbeater.cc:511] Master 127.0.98.61:41339 requested a full tablet report, sending...
I20260926 11:30:14.424296 7454 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39305
I20260926 11:30:14.424611 7454 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:14.424670 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.405783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59005 (local address 127.0.98.62:33195)
0926 11:30:14.405911 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:14.405916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:14.405952 (+ 36us) server_negotiation.cc:407] Connection header received
0926 11:30:14.406356 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:14.406359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:14.406423 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:14.406497 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:14.408430 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.409237 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.410774 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.411184 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.411278 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.420757 (+ 9479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:14.420762 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:14.420765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:14.420799 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:14.422517 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.422994 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.423000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.423004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.423087 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.423437 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.423443 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.423448 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.423679 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:14.423825 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:14.424398 (+ 573us) server_negotiation.cc:299] Negotiation successful
0926 11:30:14.424515 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 11:30:14.424762 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.405602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.98.1:59005)
0926 11:30:14.405880 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:14.405894 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:14.406170 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:14.406649 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:14.406655 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:14.407515 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:14.408248 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.408267 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.409402 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.409406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.410633 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.410643 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.418492 (+ 7849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.418496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.418603 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.419035 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:14.419052 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:14.420591 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:14.422622 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.422627 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.422629 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.422872 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.423189 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.423191 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.423193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.423294 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.423870 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:14.423876 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:14.423995 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 11:30:14.424105 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 11:30:14.424266 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":228,"spinlock_wait_cycles":14848,"thread_start_us":93,"threads_started":1}
I20260926 11:30:14.424788 7454 heartbeater.cc:511] Master 127.0.98.60:39305 requested a full tablet report, sending...
I20260926 11:30:14.425374 7084 ts_manager.cc:194] Registered new tserver with Master: 4b3cc5c906eb42cca07bc2854fc9ff3c (127.0.98.1:46203)
I20260926 11:30:14.425410 7455 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33195
I20260926 11:30:14.425494 7455 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:14.425666 7455 heartbeater.cc:511] Master 127.0.98.62:33195 requested a full tablet report, sending...
I20260926 11:30:14.425731 7275 ts_manager.cc:194] Registered new tserver with Master: 4b3cc5c906eb42cca07bc2854fc9ff3c (127.0.98.1:46203)
I20260926 11:30:14.426493 6492 ts_manager.cc:194] Re-registered known tserver with Master: 4b3cc5c906eb42cca07bc2854fc9ff3c (127.0.98.1:46203)
I20260926 11:30:14.427348 6492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59005
W20260926 11:30:14.531678 7462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:14.531983 7462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:14.532063 7462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:14.535992 7462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:14.536047 7462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:14.536134 7462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:30:14.541455 7462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42249
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44247
--tserver_master_addrs=127.0.98.62:33195,127.0.98.61:41339,127.0.98.60:39305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:14.542526 7462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:14.543535 7462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:14.552726 7467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:14.552699 7470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:14.552759 7468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:14.552928 7462 server_base.cc:1034] running on GCE node
I20260926 11:30:14.553500 7462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:14.554205 7462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:14.555384 7462 hybrid_clock.cc:648] HybridClock initialized: now 1790422214555350 us; error 46 us; skew 500 ppm
Sep 26 11:30:14 dist-test-slave-mrck krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422214, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:14.557929 7462 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 11:30:14.558727 7462 webserver.cc:460] Webserver started at http://127.0.98.2:44247/ using document root <none> and password file <none>
I20260926 11:30:14.559257 7462 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:14.559376 7462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:14.563064 7462 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:30:14.565078 7477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:14.566363 7462 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 11:30:14.566495 7462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "26d2d13738244b0399f95e5e8a836d82"
format_stamp: "Formatted at 2026-09-26 11:30:13 on dist-test-slave-mrck"
server_key: "6f2e6ede0f233ce89926bb49793a0621"
server_key_iv: "4b8808bf78cae1ed32dab590f8cd54b5"
server_key_version: "encryptionkey@0"
I20260926 11:30:14.567024 7462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:14.591176 6951 heartbeater.cc:503] Master 127.0.98.62:33195 was elected leader, sending a full tablet report...
I20260926 11:30:14.608008 7462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:14.611379 7462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:14.611583 7462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:14.612392 7462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:14.613626 7462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:14.613685 7462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:14.613752 7462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:14.613790 7462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:14.623163 7462 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42249
I20260926 11:30:14.623188 7590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42249 every 8 connection(s)
I20260926 11:30:14.624212 7462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:30:14.627399 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 7462
I20260926 11:30:14.627562 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 6822
I20260926 11:30:14.636569 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46037
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33195
--webserver_port=36463
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:33195,127.0.98.61:41339
--tserver_master_addrs=127.0.98.62:33195,127.0.98.61:41339,127.0.98.60:39305 with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 11:30:14 dist-test-slave-mrck krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422214, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 26 11:30:14 dist-test-slave-mrck krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422214, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 26 11:30:14 dist-test-slave-mrck krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422214, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 11:30:14.642133 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.626596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49663 (local address 127.0.98.60:39305)
0926 11:30:14.626749 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:14.626754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:14.627133 (+ 379us) server_negotiation.cc:407] Connection header received
0926 11:30:14.628479 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:14.628484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:14.628535 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:14.628588 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:14.630647 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.631572 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.633601 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.633918 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.634001 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.637674 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:14.637681 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:14.637686 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:14.637724 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:14.639509 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.640192 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.640198 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.640203 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.640287 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.640713 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.640717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.640718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.640997 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:14.641093 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:14.641836 (+ 743us) server_negotiation.cc:299] Negotiation successful
0926 11:30:14.641979 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 11:30:14.642912 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.628455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48701 (local address 127.0.98.62:33195)
0926 11:30:14.630197 (+ 1742us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:14.630202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:14.630216 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:30:14.630274 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:14.630280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:14.630331 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:14.630412 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:14.631331 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.632083 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.633585 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.633916 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.633991 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.638521 (+ 4530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:14.638526 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:14.638530 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:14.638561 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:14.640499 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.641018 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.641023 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.641028 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.641117 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.641420 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.641423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.641424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.641620 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:14.641712 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:14.642669 (+ 957us) server_negotiation.cc:299] Negotiation successful
0926 11:30:14.642781 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1648}
I20260926 11:30:14.642966 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.626054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56871 (local address 127.0.98.61:41339)
0926 11:30:14.626182 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:14.626187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:14.627499 (+ 1312us) server_negotiation.cc:407] Connection header received
0926 11:30:14.628480 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:14.628484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:14.628535 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:14.628588 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:14.630630 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.631460 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.633585 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.633925 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.633991 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.639061 (+ 5070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:14.639068 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:14.639072 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:14.639107 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:14.640732 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.641183 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.641190 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.641194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.641279 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.641585 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.641591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.641596 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.641859 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:14.641946 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:14.642581 (+ 635us) server_negotiation.cc:299] Negotiation successful
0926 11:30:14.642719 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 11:30:14.643684 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.627468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.98.2:48701)
0926 11:30:14.629700 (+ 2232us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:14.629713 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:14.629804 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:14.630867 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:14.630870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:14.631052 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:14.631183 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.631190 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.632213 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.632217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.633464 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.633470 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.634184 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.634188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.634459 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.635326 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:14.635337 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:14.638352 (+ 3015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:14.640640 (+ 2288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.640643 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.640648 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.640918 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.641216 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.641219 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.641221 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.641316 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.641757 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:14.641760 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:14.641840 (+ 80us) client_negotiation.cc:769] Sending connection context
0926 11:30:14.641950 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 11:30:14.642297 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2165,"mutex_wait_us":336,"spinlock_wait_cycles":1792,"thread_start_us":76,"threads_started":1}
I20260926 11:30:14.644057 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.626464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39305 (local address 127.0.98.2:49663)
0926 11:30:14.626968 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:14.627019 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:14.628241 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:14.628978 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:14.628986 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:14.629612 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:14.630474 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.630493 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.631725 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.631730 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.633463 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.633477 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.634185 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.634189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.634431 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.635463 (+ 1032us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:14.635483 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:14.637440 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:14.639644 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.639652 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.639669 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.640070 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.640407 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.640410 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.640412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.640593 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.641150 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:14.641159 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:14.641559 (+ 400us) client_negotiation.cc:769] Sending connection context
0926 11:30:14.642333 (+ 774us) client_negotiation.cc:240] Negotiation successful
0926 11:30:14.642442 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":327,"spinlock_wait_cycles":5504,"thread_start_us":156,"threads_started":1}
I20260926 11:30:14.644744 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.626464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41339 (local address 127.0.98.2:56871)
0926 11:30:14.627416 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:14.627432 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:14.628241 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:14.628978 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:14.628986 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:14.629594 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:14.630471 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.630494 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.631619 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.631622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.633447 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.633458 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.634184 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.634188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.634462 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.635278 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:14.635307 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:14.638905 (+ 3598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:14.640843 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.640846 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.640849 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.641076 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.641390 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.641396 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.641401 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.641488 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.642008 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:14.642013 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:14.642115 (+ 102us) client_negotiation.cc:769] Sending connection context
0926 11:30:14.643608 (+ 1493us) client_negotiation.cc:240] Negotiation successful
0926 11:30:14.644536 (+ 928us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"mutex_wait_us":535,"spinlock_wait_cycles":22272,"thread_start_us":146,"threads_started":1}
I20260926 11:30:14.645215 7591 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39305
I20260926 11:30:14.645367 7592 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33195
I20260926 11:30:14.645465 7592 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:14.645465 7591 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:14.645534 7593 heartbeater.cc:347] Connected to a master server at 127.0.98.61:41339
I20260926 11:30:14.645622 7593 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:14.646137 7593 heartbeater.cc:511] Master 127.0.98.61:41339 requested a full tablet report, sending...
I20260926 11:30:14.646135 7591 heartbeater.cc:511] Master 127.0.98.60:39305 requested a full tablet report, sending...
I20260926 11:30:14.646135 7592 heartbeater.cc:511] Master 127.0.98.62:33195 requested a full tablet report, sending...
I20260926 11:30:14.647246 6492 ts_manager.cc:194] Re-registered known tserver with Master: 26d2d13738244b0399f95e5e8a836d82 (127.0.98.2:42249)
I20260926 11:30:14.647384 7084 ts_manager.cc:194] Registered new tserver with Master: 26d2d13738244b0399f95e5e8a836d82 (127.0.98.2:42249)
I20260926 11:30:14.648049 7275 ts_manager.cc:194] Registered new tserver with Master: 26d2d13738244b0399f95e5e8a836d82 (127.0.98.2:42249)
I20260926 11:30:14.648068 6492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48701
I20260926 11:30:14.713742 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.709635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39305 (local address 127.0.0.1:46616)
0926 11:30:14.709874 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:14.709887 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:14.709980 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:14.710288 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:14.710290 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:14.710303 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:30:14.710432 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.710439 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.711231 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.711238 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.712087 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.712097 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.713172 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.713177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.713331 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.713357 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 11:30:14.713476 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 11:30:14.713586 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260926 11:30:14.713765 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.709756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46616 (local address 127.0.98.60:39305)
0926 11:30:14.709900 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:14.709905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:14.709933 (+ 28us) server_negotiation.cc:407] Connection header received
0926 11:30:14.710089 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:14.710092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:14.710155 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:14.710272 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:30:14.710571 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.711109 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.712214 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.713016 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.713141 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.713288 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:14.713491 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 11:30:14.713606 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 11:30:14.714843 7285 raft_consensus.cc:1203] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 11:30:14.715448 7317 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f281a9b384774f93b3ca169c35839d66. Latest consensus state: current_term: 1 leader_uuid: "f281a9b384774f93b3ca169c35839d66" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: true } } }
I20260926 11:30:14.715590 7317 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [sys.catalog]: This master's current role is: LEARNER
W20260926 11:30:14.754597 7598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:14.754830 7598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:14.754871 7598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:14.758663 7598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:14.758735 7598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:14.758791 7598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:30:14.763911 7598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46037
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36463
--tserver_master_addrs=127.0.98.62:33195,127.0.98.61:41339,127.0.98.60:39305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:14.764909 7598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:14.765889 7598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:14.775594 7604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:14.775595 7606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:14.775594 7603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:14.775905 7598 server_base.cc:1034] running on GCE node
I20260926 11:30:14.776247 7598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:14.776870 7598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:14.778026 7598 hybrid_clock.cc:648] HybridClock initialized: now 1790422214778012 us; error 27 us; skew 500 ppm
Sep 26 11:30:14 dist-test-slave-mrck krb5kdc[6447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422214, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:14.780912 7598 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 11:30:14.781955 7598 webserver.cc:460] Webserver started at http://127.0.98.3:36463/ using document root <none> and password file <none>
I20260926 11:30:14.782522 7598 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:14.782605 7598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:14.786764 7598 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 11:30:14.788859 7613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:14.790081 7598 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:30:14.790196 7598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "5fb677408a224fa2b4a8eee4dc918823"
format_stamp: "Formatted at 2026-09-26 11:30:13 on dist-test-slave-mrck"
server_key: "f4ed163dc956f1600b2e8c8f01e9257a"
server_key_iv: "a7822abda27a8f85348dc4e7323d9dca"
server_key_version: "encryptionkey@0"
I20260926 11:30:14.790719 7598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:14.802722 7598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:14.806015 7598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:14.806223 7598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:14.806798 7598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:14.807614 7598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:14.807657 7598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:14.807762 7598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:14.807798 7598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:14.815999 7598 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46037
I20260926 11:30:14.816063 7726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46037 every 8 connection(s)
I20260926 11:30:14.816989 7598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:30:14.825721 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 7598
Sep 26 11:30:14 dist-test-slave-mrck krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422214, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 26 11:30:14 dist-test-slave-mrck krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422214, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 26 11:30:14 dist-test-slave-mrck krb5kdc[6447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422214, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 11:30:14.833627 6531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.818780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41201 (local address 127.0.98.62:33195)
0926 11:30:14.818940 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:14.818946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:14.820343 (+ 1397us) server_negotiation.cc:407] Connection header received
0926 11:30:14.821217 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:14.821220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:14.821291 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:14.821370 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:14.822701 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.823520 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.825232 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.825590 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.825696 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.829076 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:14.829082 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:14.829089 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:14.829123 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:14.831173 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.831794 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.831798 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.831800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.831881 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.832238 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.832244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.832246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.832459 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:14.832547 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:14.833358 (+ 811us) server_negotiation.cc:299] Negotiation successful
0926 11:30:14.833466 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 11:30:14.834750 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.819045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49901 (local address 127.0.98.61:41339)
0926 11:30:14.819162 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:14.819165 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:14.819944 (+ 779us) server_negotiation.cc:407] Connection header received
0926 11:30:14.821127 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:14.821131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:14.821187 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:14.821266 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:14.822727 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.823247 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.825228 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.825858 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.826097 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.829270 (+ 3173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:14.829276 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:14.829282 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:14.829319 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:14.831711 (+ 2392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.832234 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.832237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.832239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.832335 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.832631 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.832634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.832636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.832815 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:14.832900 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:14.834434 (+ 1534us) server_negotiation.cc:299] Negotiation successful
0926 11:30:14.834566 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":45}
I20260926 11:30:14.834754 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.819257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.98.3:41201)
0926 11:30:14.820236 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:14.820249 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:14.820917 (+ 668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:14.821464 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:14.821473 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:14.822007 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:14.822554 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.822566 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.823628 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.823630 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.825126 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.825136 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.825952 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.825956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.826054 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.826682 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:14.826709 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:14.828879 (+ 2170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:14.831287 (+ 2408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.831296 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.831316 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.831660 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.831975 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.831978 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.831980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.832138 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.832597 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:14.832603 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:14.832898 (+ 295us) client_negotiation.cc:769] Sending connection context
0926 11:30:14.833078 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 11:30:14.834147 (+ 1069us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"mutex_wait_us":889,"spinlock_wait_cycles":44928,"thread_start_us":706,"threads_started":1}
I20260926 11:30:14.835665 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.819257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41339 (local address 127.0.98.3:49901)
0926 11:30:14.819795 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:14.819851 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:14.820917 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:14.821464 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:14.821472 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:14.822014 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:14.822581 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.822588 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.823394 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.823397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.825083 (+ 1686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.825093 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.826800 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.826804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.826910 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.827419 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:14.827440 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:14.829101 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:14.831854 (+ 2753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.831860 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.831864 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.832120 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.832445 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.832450 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.832451 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.832536 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.832939 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:14.832942 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:14.833024 (+ 82us) client_negotiation.cc:769] Sending connection context
0926 11:30:14.833117 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 11:30:14.833462 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":213,"spinlock_wait_cycles":11648,"thread_start_us":148,"threads_started":1}
I20260926 11:30:14.836658 7728 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33195
I20260926 11:30:14.836912 7728 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:14.837483 7728 heartbeater.cc:511] Master 127.0.98.62:33195 requested a full tablet report, sending...
I20260926 11:30:14.837927 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.820443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39305 (local address 127.0.98.3:34381)
0926 11:30:14.820734 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:14.820747 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:14.820922 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:14.821563 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:14.821565 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:14.822009 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:14.822580 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.822588 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.823663 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.823666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.824442 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.824450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.824929 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.824933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.825236 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.828466 (+ 3230us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:14.828480 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:14.829952 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:14.836202 (+ 6250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.836205 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.836207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.836444 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.836743 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:14.836748 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:14.836750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:14.836835 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:14.837334 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:14.837338 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:14.837452 (+ 114us) client_negotiation.cc:769] Sending connection context
0926 11:30:14.837551 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 11:30:14.837698 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":181,"spinlock_wait_cycles":32256,"thread_start_us":94,"threads_started":1}
I20260926 11:30:14.836663 7729 heartbeater.cc:347] Connected to a master server at 127.0.98.61:41339
I20260926 11:30:14.838064 7729 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:14.838212 7729 heartbeater.cc:511] Master 127.0.98.61:41339 requested a full tablet report, sending...
I20260926 11:30:14.838732 6492 ts_manager.cc:194] Re-registered known tserver with Master: 5fb677408a224fa2b4a8eee4dc918823 (127.0.98.3:46037)
I20260926 11:30:14.838804 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.820565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34381 (local address 127.0.98.60:39305)
0926 11:30:14.820692 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:14.820701 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:14.820837 (+ 136us) server_negotiation.cc:407] Connection header received
0926 11:30:14.821406 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:14.821409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:14.821456 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:14.821535 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:14.822728 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.823513 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.824566 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.824784 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.824896 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.830119 (+ 5223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:14.830123 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:14.830127 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:14.830156 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:14.836093 (+ 5937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.836553 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.836557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.836558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.836640 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.836944 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.836947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.836949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.837177 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:14.837275 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:14.838502 (+ 1227us) server_negotiation.cc:299] Negotiation successful
0926 11:30:14.838614 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 11:30:14.839177 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.828278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59778 (local address 127.0.98.62:33195)
0926 11:30:14.829832 (+ 1554us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:14.829836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:14.830234 (+ 398us) server_negotiation.cc:407] Connection header received
0926 11:30:14.830276 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:14.830281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:14.830345 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:14.830430 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:14.831376 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.832198 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.833307 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.833610 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.833704 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.835369 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:14.835373 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:14.835375 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:14.835398 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:14.837137 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.837616 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.837619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.837621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.837699 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.838000 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.838007 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.838009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.838205 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:14.838339 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:14.838922 (+ 583us) server_negotiation.cc:299] Negotiation successful
0926 11:30:14.839033 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1469,"thread_start_us":76,"threads_started":1}
I20260926 11:30:14.839359 7727 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39305
I20260926 11:30:14.839429 7727 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:14.839423 6492 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41201
I20260926 11:30:14.839538 7727 heartbeater.cc:511] Master 127.0.98.60:39305 requested a full tablet report, sending...
I20260926 11:30:14.840380 7084 ts_manager.cc:194] Registered new tserver with Master: 5fb677408a224fa2b4a8eee4dc918823 (127.0.98.3:46037)
I20260926 11:30:14.841282 7275 ts_manager.cc:194] Registered new tserver with Master: 5fb677408a224fa2b4a8eee4dc918823 (127.0.98.3:46037)
I20260926 11:30:14.842407 6491 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 11:30:14.845785 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.827860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59172 (local address 127.0.98.61:41339)
0926 11:30:14.831318 (+ 3458us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:14.831322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:14.831335 (+ 13us) server_negotiation.cc:407] Connection header received
0926 11:30:14.831370 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:14.831374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:14.831420 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:14.831510 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:14.832472 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.833233 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.834305 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.834899 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.835000 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.838440 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:14.838445 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:14.838448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:14.838471 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:14.843973 (+ 5502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.844482 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.844486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.844488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.844576 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.844866 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.844869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.844871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.845099 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:14.845217 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:14.845480 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 11:30:14.845609 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":231,"server-negotiator.queue_time_us":2487,"thread_start_us":93,"threads_started":1}
W20260926 11:30:14.846319 6491 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:30:14.848953 7241 sys_catalog.cc:455] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f281a9b384774f93b3ca169c35839d66" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: true } } }
I20260926 11:30:14.849084 7241 sys_catalog.cc:458] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:14.850600 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.829241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46630 (local address 127.0.98.60:39305)
0926 11:30:14.830262 (+ 1021us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:14.830266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:14.834674 (+ 4408us) server_negotiation.cc:407] Connection header received
0926 11:30:14.835576 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:14.835582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:14.835642 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:14.835707 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:14.843070 (+ 7363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.843970 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.845068 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.845412 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.845484 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.847074 (+ 1590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:14.847078 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:14.847081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:14.847105 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:14.848579 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.849067 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.849073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.849075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.849163 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:14.849485 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:14.849488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:14.849490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:14.849701 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:14.849805 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:14.850150 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 11:30:14.850426 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":951,"thread_start_us":108,"threads_started":1}
I20260926 11:30:14.852739 7746 raft_consensus.cc:1050] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66: attempting to promote NON_VOTER b5d89fa8510448b9950d2ef424068e4d to VOTER
I20260926 11:30:14.853058 7746 consensus_queue.cc:237] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: false } }
I20260926 11:30:14.854190 7094 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [term 1 FOLLOWER]: Refusing update from remote peer f281a9b384774f93b3ca169c35839d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 11:30:14.854545 7745 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 11:30:14.856011 7746 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b5d89fa8510448b9950d2ef424068e4d (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: false } } }
I20260926 11:30:14.856626 7284 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [term 1 LEARNER]: Refusing update from remote peer f281a9b384774f93b3ca169c35839d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 11:30:14.857110 7745 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 11:30:14.857497 7745 sys_catalog.cc:455] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f281a9b384774f93b3ca169c35839d66" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: false } } }
I20260926 11:30:14.857602 7745 sys_catalog.cc:458] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:14.857509 7746 sys_catalog.cc:455] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f281a9b384774f93b3ca169c35839d66" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: false } } }
I20260926 11:30:14.857929 7746 sys_catalog.cc:458] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:14.858119 7284 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b5d89fa8510448b9950d2ef424068e4d (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: false } } }
I20260926 11:30:14.859656 7317 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f281a9b384774f93b3ca169c35839d66" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: false } } }
I20260926 11:30:14.859792 7317 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:30:14.860286 7094 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, b5d89fa8510448b9950d2ef424068e4d (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: false } } }
I20260926 11:30:14.860397 7745 sys_catalog.cc:455] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f281a9b384774f93b3ca169c35839d66" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: false } } }
I20260926 11:30:14.860494 7745 sys_catalog.cc:458] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66 [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:14.861992 7755 sys_catalog.cc:455] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f281a9b384774f93b3ca169c35839d66" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: false } } }
I20260926 11:30:14.862100 7755 sys_catalog.cc:458] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:30:14.869771 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.864423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46203 (local address 127.0.0.1:44840)
0926 11:30:14.865629 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:14.865641 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:14.865724 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:14.866119 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:14.866122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:14.866139 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:30:14.866241 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.866250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.867191 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.867198 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.868091 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.868104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.869368 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.869371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.869462 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.869485 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 11:30:14.869556 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 11:30:14.869623 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1150,"thread_start_us":99,"threads_started":1}
I20260926 11:30:14.870273 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.864979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44840 (local address 127.0.98.1:46203)
0926 11:30:14.865288 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:14.865300 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:14.865704 (+ 404us) server_negotiation.cc:407] Connection header received
0926 11:30:14.865817 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:14.865821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:14.865984 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:14.866152 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:30:14.866338 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.867078 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.868231 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.869047 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.869154 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.869313 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:14.869963 (+ 650us) server_negotiation.cc:299] Negotiation successful
0926 11:30:14.870127 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":62,"threads_started":1}
I20260926 11:30:14.871099 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.865068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46037 (local address 127.0.0.1:33798)
0926 11:30:14.865889 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:14.865900 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:14.865964 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:14.866582 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:14.866584 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:14.866599 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:30:14.866681 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.866686 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.868055 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.868059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.869177 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.869186 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.870683 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.870687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.870825 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.870849 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 11:30:14.870916 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 11:30:14.870978 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"thread_start_us":71,"threads_started":1}
I20260926 11:30:14.872035 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.865295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33798 (local address 127.0.98.3:46037)
0926 11:30:14.865564 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:14.865572 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:14.866040 (+ 468us) server_negotiation.cc:407] Connection header received
0926 11:30:14.866112 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:14.866117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:14.866310 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:14.866444 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:30:14.866784 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.867943 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.869317 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.870280 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.870500 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.870633 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:14.871363 (+ 730us) server_negotiation.cc:299] Negotiation successful
0926 11:30:14.871454 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":103,"threads_started":1}
I20260926 11:30:14.873489 7388 tablet_service.cc:1467] Processing CreateTablet for tablet 5e2d3103a2f5425f940c97072083033f (DEFAULT_TABLE table=test-table [id=d8b3fc1a71c14cdb9500667cc679e41a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:30:14.874104 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.864251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42249 (local address 127.0.0.1:43578)
0926 11:30:14.864665 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:14.864679 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:14.864746 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:14.868670 (+ 3924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:14.868673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:14.868688 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:30:14.868763 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.868767 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.870461 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.870464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.871665 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:14.871673 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.873765 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.873769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:14.873870 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.873892 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 11:30:14.873934 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 11:30:14.873997 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349}
I20260926 11:30:14.874605 7388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e2d3103a2f5425f940c97072083033f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:14.874732 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:14.864454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43578 (local address 127.0.98.2:42249)
0926 11:30:14.868229 (+ 3775us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:14.868240 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:14.868260 (+ 20us) server_negotiation.cc:407] Connection header received
0926 11:30:14.868330 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:14.868335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:14.868522 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:14.868901 (+ 379us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:30:14.869019 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.870269 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.872034 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:14.873638 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:14.874318 (+ 680us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:14.874455 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:14.874499 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 11:30:14.874578 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3671,"thread_start_us":114,"threads_started":1}
I20260926 11:30:14.874644 7661 tablet_service.cc:1467] Processing CreateTablet for tablet 5e2d3103a2f5425f940c97072083033f (DEFAULT_TABLE table=test-table [id=d8b3fc1a71c14cdb9500667cc679e41a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:30:14.875584 7661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e2d3103a2f5425f940c97072083033f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:14.877027 7525 tablet_service.cc:1467] Processing CreateTablet for tablet 5e2d3103a2f5425f940c97072083033f (DEFAULT_TABLE table=test-table [id=d8b3fc1a71c14cdb9500667cc679e41a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:30:14.877825 7525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e2d3103a2f5425f940c97072083033f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:14.882980 7761 tablet_bootstrap.cc:492] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c: Bootstrap starting.
I20260926 11:30:14.884531 7762 tablet_bootstrap.cc:492] T 5e2d3103a2f5425f940c97072083033f P 5fb677408a224fa2b4a8eee4dc918823: Bootstrap starting.
I20260926 11:30:14.885049 7763 tablet_bootstrap.cc:492] T 5e2d3103a2f5425f940c97072083033f P 26d2d13738244b0399f95e5e8a836d82: Bootstrap starting.
I20260926 11:30:14.885789 7761 tablet_bootstrap.cc:654] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:14.886499 7761 log.cc:824] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c: Log is configured to *not* fsync() on all Append() calls
I20260926 11:30:14.886957 7762 tablet_bootstrap.cc:654] T 5e2d3103a2f5425f940c97072083033f P 5fb677408a224fa2b4a8eee4dc918823: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:14.887547 7763 tablet_bootstrap.cc:654] T 5e2d3103a2f5425f940c97072083033f P 26d2d13738244b0399f95e5e8a836d82: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:14.887820 7762 log.cc:824] T 5e2d3103a2f5425f940c97072083033f P 5fb677408a224fa2b4a8eee4dc918823: Log is configured to *not* fsync() on all Append() calls
I20260926 11:30:14.888361 7763 log.cc:824] T 5e2d3103a2f5425f940c97072083033f P 26d2d13738244b0399f95e5e8a836d82: Log is configured to *not* fsync() on all Append() calls
I20260926 11:30:14.888516 7761 tablet_bootstrap.cc:492] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c: No bootstrap required, opened a new log
I20260926 11:30:14.888720 7761 ts_tablet_manager.cc:1397] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 11:30:14.890085 7762 tablet_bootstrap.cc:492] T 5e2d3103a2f5425f940c97072083033f P 5fb677408a224fa2b4a8eee4dc918823: No bootstrap required, opened a new log
I20260926 11:30:14.890256 7762 ts_tablet_manager.cc:1397] T 5e2d3103a2f5425f940c97072083033f P 5fb677408a224fa2b4a8eee4dc918823: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 11:30:14.893030 7763 tablet_bootstrap.cc:492] T 5e2d3103a2f5425f940c97072083033f P 26d2d13738244b0399f95e5e8a836d82: No bootstrap required, opened a new log
I20260926 11:30:14.893227 7763 ts_tablet_manager.cc:1397] T 5e2d3103a2f5425f940c97072083033f P 26d2d13738244b0399f95e5e8a836d82: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260926 11:30:14.893253 7761 raft_consensus.cc:348] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d2d13738244b0399f95e5e8a836d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42249 } } peers { permanent_uuid: "4b3cc5c906eb42cca07bc2854fc9ff3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46203 } } peers { permanent_uuid: "5fb677408a224fa2b4a8eee4dc918823" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46037 } }
I20260926 11:30:14.893522 7761 raft_consensus.cc:374] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:14.893575 7761 raft_consensus.cc:729] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b3cc5c906eb42cca07bc2854fc9ff3c, State: Initialized, Role: FOLLOWER
I20260926 11:30:14.894037 7761 consensus_queue.cc:260] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d2d13738244b0399f95e5e8a836d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42249 } } peers { permanent_uuid: "4b3cc5c906eb42cca07bc2854fc9ff3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46203 } } peers { permanent_uuid: "5fb677408a224fa2b4a8eee4dc918823" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46037 } }
I20260926 11:30:14.894071 7762 raft_consensus.cc:348] T 5e2d3103a2f5425f940c97072083033f P 5fb677408a224fa2b4a8eee4dc918823 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d2d13738244b0399f95e5e8a836d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42249 } } peers { permanent_uuid: "4b3cc5c906eb42cca07bc2854fc9ff3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46203 } } peers { permanent_uuid: "5fb677408a224fa2b4a8eee4dc918823" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46037 } }
I20260926 11:30:14.894299 7762 raft_consensus.cc:374] T 5e2d3103a2f5425f940c97072083033f P 5fb677408a224fa2b4a8eee4dc918823 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:14.894369 7762 raft_consensus.cc:729] T 5e2d3103a2f5425f940c97072083033f P 5fb677408a224fa2b4a8eee4dc918823 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fb677408a224fa2b4a8eee4dc918823, State: Initialized, Role: FOLLOWER
I20260926 11:30:14.894829 7762 consensus_queue.cc:260] T 5e2d3103a2f5425f940c97072083033f P 5fb677408a224fa2b4a8eee4dc918823 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d2d13738244b0399f95e5e8a836d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42249 } } peers { permanent_uuid: "4b3cc5c906eb42cca07bc2854fc9ff3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46203 } } peers { permanent_uuid: "5fb677408a224fa2b4a8eee4dc918823" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46037 } }
I20260926 11:30:14.895025 7761 ts_tablet_manager.cc:1428] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 11:30:14.895401 7455 heartbeater.cc:503] Master 127.0.98.62:33195 was elected leader, sending a full tablet report...
I20260926 11:30:14.895759 7762 ts_tablet_manager.cc:1428] T 5e2d3103a2f5425f940c97072083033f P 5fb677408a224fa2b4a8eee4dc918823: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
I20260926 11:30:14.896988 7728 heartbeater.cc:503] Master 127.0.98.62:33195 was elected leader, sending a full tablet report...
I20260926 11:30:14.900810 7763 raft_consensus.cc:348] T 5e2d3103a2f5425f940c97072083033f P 26d2d13738244b0399f95e5e8a836d82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d2d13738244b0399f95e5e8a836d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42249 } } peers { permanent_uuid: "4b3cc5c906eb42cca07bc2854fc9ff3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46203 } } peers { permanent_uuid: "5fb677408a224fa2b4a8eee4dc918823" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46037 } }
I20260926 11:30:14.901080 7763 raft_consensus.cc:374] T 5e2d3103a2f5425f940c97072083033f P 26d2d13738244b0399f95e5e8a836d82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:14.901136 7763 raft_consensus.cc:729] T 5e2d3103a2f5425f940c97072083033f P 26d2d13738244b0399f95e5e8a836d82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d2d13738244b0399f95e5e8a836d82, State: Initialized, Role: FOLLOWER
I20260926 11:30:14.901587 7763 consensus_queue.cc:260] T 5e2d3103a2f5425f940c97072083033f P 26d2d13738244b0399f95e5e8a836d82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d2d13738244b0399f95e5e8a836d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42249 } } peers { permanent_uuid: "4b3cc5c906eb42cca07bc2854fc9ff3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46203 } } peers { permanent_uuid: "5fb677408a224fa2b4a8eee4dc918823" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46037 } }
I20260926 11:30:14.902488 7763 ts_tablet_manager.cc:1428] T 5e2d3103a2f5425f940c97072083033f P 26d2d13738244b0399f95e5e8a836d82: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20260926 11:30:14.902803 7592 heartbeater.cc:503] Master 127.0.98.62:33195 was elected leader, sending a full tablet report...
W20260926 11:30:14.903916 7457 tablet.cc:2369] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:30:15.062929 7767 raft_consensus.cc:482] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:30:15.063164 7767 raft_consensus.cc:504] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d2d13738244b0399f95e5e8a836d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42249 } } peers { permanent_uuid: "4b3cc5c906eb42cca07bc2854fc9ff3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46203 } } peers { permanent_uuid: "5fb677408a224fa2b4a8eee4dc918823" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46037 } }
I20260926 11:30:15.064400 7767 leader_election.cc:288] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26d2d13738244b0399f95e5e8a836d82 (127.0.98.2:42249), 5fb677408a224fa2b4a8eee4dc918823 (127.0.98.3:46037)
I20260926 11:30:15.068220 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:15.065013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53413 (local address 127.0.98.3:46037)
0926 11:30:15.065163 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:15.065169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:15.065184 (+ 15us) server_negotiation.cc:407] Connection header received
0926 11:30:15.065235 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:15.065237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:15.065274 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:15.065337 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:30:15.065648 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:15.066121 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:15.067219 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:15.067609 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:15.067689 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:15.067805 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:15.068011 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 11:30:15.068088 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 11:30:15.068220 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:15.064889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57449 (local address 127.0.98.2:42249)
0926 11:30:15.065008 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:15.065015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:15.065032 (+ 17us) server_negotiation.cc:407] Connection header received
0926 11:30:15.065215 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:15.065218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:15.065280 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:15.065375 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:30:15.065613 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:15.066114 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:15.067243 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:15.067636 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:15.067794 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:15.067858 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:15.068039 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 11:30:15.068110 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 11:30:15.068243 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:15.064665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46037 (local address 127.0.98.1:53413)
0926 11:30:15.064945 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:15.064962 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:15.065066 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:15.065372 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:15.065375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:15.065391 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:30:15.065513 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:15.065519 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:15.066249 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:15.066254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:15.067091 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:15.067099 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:15.067753 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:15.067766 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:15.067859 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:15.067890 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 11:30:15.067974 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 11:30:15.068077 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":3,"thread_start_us":84,"threads_started":1}
I20260926 11:30:15.068243 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:15.064710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42249 (local address 127.0.98.1:57449)
0926 11:30:15.064967 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:15.064981 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:15.065083 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:15.065398 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:15.065401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:15.065418 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:30:15.065509 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:15.065514 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:15.066221 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:15.066224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:15.067094 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:15.067103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:15.067805 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:15.067808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:15.067895 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:15.067912 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 11:30:15.068011 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 11:30:15.068089 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":56,"spinlock_wait_cycles":2432,"thread_start_us":73,"threads_started":1}
W20260926 11:30:15.068826 7730 tablet.cc:2369] T 5e2d3103a2f5425f940c97072083033f P 5fb677408a224fa2b4a8eee4dc918823: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:30:15.069239 7545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e2d3103a2f5425f940c97072083033f" candidate_uuid: "4b3cc5c906eb42cca07bc2854fc9ff3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d2d13738244b0399f95e5e8a836d82" is_pre_election: true
I20260926 11:30:15.069603 7545 raft_consensus.cc:2454] T 5e2d3103a2f5425f940c97072083033f P 26d2d13738244b0399f95e5e8a836d82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b3cc5c906eb42cca07bc2854fc9ff3c in term 0.
I20260926 11:30:15.069547 7681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e2d3103a2f5425f940c97072083033f" candidate_uuid: "4b3cc5c906eb42cca07bc2854fc9ff3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fb677408a224fa2b4a8eee4dc918823" is_pre_election: true
I20260926 11:30:15.069849 7681 raft_consensus.cc:2454] T 5e2d3103a2f5425f940c97072083033f P 5fb677408a224fa2b4a8eee4dc918823 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b3cc5c906eb42cca07bc2854fc9ff3c in term 0.
I20260926 11:30:15.070073 7341 leader_election.cc:302] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d2d13738244b0399f95e5e8a836d82, 4b3cc5c906eb42cca07bc2854fc9ff3c; no voters:
I20260926 11:30:15.070395 7767 raft_consensus.cc:2781] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:30:15.070484 7767 raft_consensus.cc:482] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:30:15.070526 7767 raft_consensus.cc:3037] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:15.071738 7767 raft_consensus.cc:504] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d2d13738244b0399f95e5e8a836d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42249 } } peers { permanent_uuid: "4b3cc5c906eb42cca07bc2854fc9ff3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46203 } } peers { permanent_uuid: "5fb677408a224fa2b4a8eee4dc918823" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46037 } }
I20260926 11:30:15.072213 7767 leader_election.cc:288] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c [CANDIDATE]: Term 1 election: Requested vote from peers 26d2d13738244b0399f95e5e8a836d82 (127.0.98.2:42249), 5fb677408a224fa2b4a8eee4dc918823 (127.0.98.3:46037)
I20260926 11:30:15.072561 7545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e2d3103a2f5425f940c97072083033f" candidate_uuid: "4b3cc5c906eb42cca07bc2854fc9ff3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d2d13738244b0399f95e5e8a836d82"
I20260926 11:30:15.072674 7545 raft_consensus.cc:3037] T 5e2d3103a2f5425f940c97072083033f P 26d2d13738244b0399f95e5e8a836d82 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:15.073737 7545 raft_consensus.cc:2454] T 5e2d3103a2f5425f940c97072083033f P 26d2d13738244b0399f95e5e8a836d82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b3cc5c906eb42cca07bc2854fc9ff3c in term 1.
I20260926 11:30:15.073978 7681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e2d3103a2f5425f940c97072083033f" candidate_uuid: "4b3cc5c906eb42cca07bc2854fc9ff3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fb677408a224fa2b4a8eee4dc918823"
I20260926 11:30:15.074014 7341 leader_election.cc:302] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d2d13738244b0399f95e5e8a836d82, 4b3cc5c906eb42cca07bc2854fc9ff3c; no voters:
I20260926 11:30:15.074111 7681 raft_consensus.cc:3037] T 5e2d3103a2f5425f940c97072083033f P 5fb677408a224fa2b4a8eee4dc918823 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:15.074179 7767 raft_consensus.cc:2781] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:30:15.074388 7767 raft_consensus.cc:686] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c [term 1 LEADER]: Becoming Leader. State: Replica: 4b3cc5c906eb42cca07bc2854fc9ff3c, State: Running, Role: LEADER
I20260926 11:30:15.074754 7767 consensus_queue.cc:237] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d2d13738244b0399f95e5e8a836d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42249 } } peers { permanent_uuid: "4b3cc5c906eb42cca07bc2854fc9ff3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46203 } } peers { permanent_uuid: "5fb677408a224fa2b4a8eee4dc918823" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46037 } }
I20260926 11:30:15.075393 7681 raft_consensus.cc:2454] T 5e2d3103a2f5425f940c97072083033f P 5fb677408a224fa2b4a8eee4dc918823 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b3cc5c906eb42cca07bc2854fc9ff3c in term 1.
I20260926 11:30:15.078423 6492 catalog_manager.cc:5495] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b3cc5c906eb42cca07bc2854fc9ff3c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4b3cc5c906eb42cca07bc2854fc9ff3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26d2d13738244b0399f95e5e8a836d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b3cc5c906eb42cca07bc2854fc9ff3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fb677408a224fa2b4a8eee4dc918823" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46037 } health_report { overall_health: UNKNOWN } } }
I20260926 11:30:15.113570 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:15.109855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44848 (local address 127.0.98.1:46203)
0926 11:30:15.109993 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:15.110000 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:15.110022 (+ 22us) server_negotiation.cc:407] Connection header received
0926 11:30:15.110124 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:15.110128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:15.110218 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:15.110344 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:30:15.110593 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:15.111156 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:15.112171 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:15.112401 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:15.112488 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:15.112784 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:30:15.113228 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:30:15.113339 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:15.113402 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 11:30:15.113456 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 11:30:15.119982 7681 raft_consensus.cc:1261] T 5e2d3103a2f5425f940c97072083033f P 5fb677408a224fa2b4a8eee4dc918823 [term 1 FOLLOWER]: Refusing update from remote peer 4b3cc5c906eb42cca07bc2854fc9ff3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:30:15.119982 7545 raft_consensus.cc:1261] T 5e2d3103a2f5425f940c97072083033f P 26d2d13738244b0399f95e5e8a836d82 [term 1 FOLLOWER]: Refusing update from remote peer 4b3cc5c906eb42cca07bc2854fc9ff3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:30:15.120870 7767 consensus_queue.cc:1035] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d2d13738244b0399f95e5e8a836d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:30:15.121136 7772 consensus_queue.cc:1035] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fb677408a224fa2b4a8eee4dc918823" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 11:30:15.126499 7594 tablet.cc:2369] T 5e2d3103a2f5425f940c97072083033f P 26d2d13738244b0399f95e5e8a836d82: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:30:15.129562 7775 mvcc.cc:205] Tried to move back new op lower bound from 7333569393123201024 to 7333569392949440512. Current Snapshot: MvccSnapshot[applied={T|T < 7333569393123201024}]
I20260926 11:30:15.131073 7777 mvcc.cc:205] Tried to move back new op lower bound from 7333569393123201024 to 7333569392949440512. Current Snapshot: MvccSnapshot[applied={T|T < 7333569393123201024}]
I20260926 11:30:15.132144 7776 mvcc.cc:205] Tried to move back new op lower bound from 7333569393123201024 to 7333569392949440512. Current Snapshot: MvccSnapshot[applied={T|T < 7333569393123201024}]
I20260926 11:30:15.140650 6491 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:30:15.140808 6491 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:30:15.146159 6491 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f281a9b384774f93b3ca169c35839d66: Sending DeleteTablet for 3 replicas of tablet 5e2d3103a2f5425f940c97072083033f
I20260926 11:30:15.146890 7388 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e2d3103a2f5425f940c97072083033f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:30:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44840
I20260926 11:30:15.146883 7525 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e2d3103a2f5425f940c97072083033f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:30:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43578
I20260926 11:30:15.147100 7661 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e2d3103a2f5425f940c97072083033f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:30:15 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33798
I20260926 11:30:15.147475 7786 tablet_replica.cc:330] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c: stopping tablet replica
I20260926 11:30:15.147588 7787 tablet_replica.cc:330] T 5e2d3103a2f5425f940c97072083033f P 5fb677408a224fa2b4a8eee4dc918823: stopping tablet replica
I20260926 11:30:15.147948 7786 raft_consensus.cc:2229] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:30:15.147949 7787 raft_consensus.cc:2229] T 5e2d3103a2f5425f940c97072083033f P 5fb677408a224fa2b4a8eee4dc918823 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:30:15.148288 7787 raft_consensus.cc:2258] T 5e2d3103a2f5425f940c97072083033f P 5fb677408a224fa2b4a8eee4dc918823 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:30:15.148581 7786 raft_consensus.cc:2258] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:30:15.149969 7787 ts_tablet_manager.cc:1905] T 5e2d3103a2f5425f940c97072083033f P 5fb677408a224fa2b4a8eee4dc918823: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:30:15.150609 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 6460
I20260926 11:30:15.150615 7786 ts_tablet_manager.cc:1905] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:30:15.153924 7787 ts_tablet_manager.cc:1918] T 5e2d3103a2f5425f940c97072083033f P 5fb677408a224fa2b4a8eee4dc918823: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:30:15.154032 7787 log.cc:1196] T 5e2d3103a2f5425f940c97072083033f P 5fb677408a224fa2b4a8eee4dc918823: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/wal/wals/5e2d3103a2f5425f940c97072083033f
I20260926 11:30:15.154382 7787 ts_tablet_manager.cc:1939] T 5e2d3103a2f5425f940c97072083033f P 5fb677408a224fa2b4a8eee4dc918823: Deleting consensus metadata
I20260926 11:30:15.154606 7786 ts_tablet_manager.cc:1918] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:30:15.154747 7786 log.cc:1196] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-0/wal/wals/5e2d3103a2f5425f940c97072083033f
I20260926 11:30:15.155121 7786 ts_tablet_manager.cc:1939] T 5e2d3103a2f5425f940c97072083033f P 4b3cc5c906eb42cca07bc2854fc9ff3c: Deleting consensus metadata
W20260926 11:30:15.158600 7615 connection.cc:535] server connection from 127.0.0.1:33798 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 11:30:15.159005 7344 connection.cc:535] server connection from 127.0.0.1:44840 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 11:30:15.160276 7785 tablet_replica.cc:330] T 5e2d3103a2f5425f940c97072083033f P 26d2d13738244b0399f95e5e8a836d82: stopping tablet replica
I20260926 11:30:15.160640 7785 raft_consensus.cc:2229] T 5e2d3103a2f5425f940c97072083033f P 26d2d13738244b0399f95e5e8a836d82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:30:15.161166 7785 raft_consensus.cc:2258] T 5e2d3103a2f5425f940c97072083033f P 26d2d13738244b0399f95e5e8a836d82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:30:15.162873 7785 ts_tablet_manager.cc:1905] T 5e2d3103a2f5425f940c97072083033f P 26d2d13738244b0399f95e5e8a836d82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:30:15.165920 7785 ts_tablet_manager.cc:1918] T 5e2d3103a2f5425f940c97072083033f P 26d2d13738244b0399f95e5e8a836d82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:30:15.166014 7785 log.cc:1196] T 5e2d3103a2f5425f940c97072083033f P 26d2d13738244b0399f95e5e8a836d82: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/wal/wals/5e2d3103a2f5425f940c97072083033f
I20260926 11:30:15.166311 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:15.149249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59186 (local address 127.0.98.61:41339)
0926 11:30:15.149638 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:15.149642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:15.155856 (+ 6214us) server_negotiation.cc:407] Connection header received
0926 11:30:15.156067 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:15.156073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:15.156136 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:15.156195 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:15.157354 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:15.158164 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:15.159622 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:15.160015 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:15.160117 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:15.162252 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:15.162269 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:15.162272 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:15.162301 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:15.164419 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:15.164916 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:15.164920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:15.164922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:15.165006 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:15.165336 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:15.165340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:15.165342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:15.165548 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:15.165647 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:15.165992 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 11:30:15.166114 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315}
I20260926 11:30:15.166325 7785 ts_tablet_manager.cc:1939] T 5e2d3103a2f5425f940c97072083033f P 26d2d13738244b0399f95e5e8a836d82: Deleting consensus metadata
W20260926 11:30:15.167521 7478 connection.cc:441] server connection from 127.0.0.1:43578 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43578 (request call id 1) could send its response
I20260926 11:30:15.173678 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:15.162797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59188 (local address 127.0.98.61:41339)
0926 11:30:15.162913 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:15.162917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:15.163835 (+ 918us) server_negotiation.cc:407] Connection header received
0926 11:30:15.164002 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:15.164005 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:15.164047 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:15.164096 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:15.165473 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:15.166236 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:15.167354 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:15.167662 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:15.167796 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:15.169600 (+ 1804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:15.169606 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:15.169609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:15.169634 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:15.171439 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:15.172057 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:15.172061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:15.172062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:15.172142 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:15.172514 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:15.172517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:15.172519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:15.172753 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:15.172877 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:15.173305 (+ 428us) server_negotiation.cc:299] Negotiation successful
0926 11:30:15.173457 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":48}
I20260926 11:30:15.177977 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:15.149349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46642 (local address 127.0.98.60:39305)
0926 11:30:15.149459 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:15.149464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:15.168010 (+ 18546us) server_negotiation.cc:407] Connection header received
0926 11:30:15.168087 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:15.168090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:15.168151 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:15.168245 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:15.169215 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:15.170023 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:15.171404 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:15.171820 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:15.171912 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:15.174174 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:15.174179 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:15.174182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:15.174209 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:15.176157 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:15.176687 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:15.176691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:15.176692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:15.176775 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:15.177065 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:15.177070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:15.177072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:15.177275 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:15.177428 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:15.177675 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 11:30:15.177805 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 11:30:15.177979 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:15.163277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46644 (local address 127.0.98.60:39305)
0926 11:30:15.163402 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:15.163409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:15.168725 (+ 5316us) server_negotiation.cc:407] Connection header received
0926 11:30:15.168904 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:15.168911 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:15.168966 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:15.169025 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:15.169950 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:15.170796 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:15.172072 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:15.172410 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:15.172499 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:15.174115 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:15.174119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:15.174122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:15.174145 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:15.176298 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:15.176730 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:15.176734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:15.176736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:15.176817 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:15.177169 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:15.177173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:15.177174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:15.177322 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:15.177428 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:15.177782 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 11:30:15.177860 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 11:30:16.132889 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:16.132450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.98.3:55433)
0926 11:30:16.132715 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:16.132794 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":101,"threads_started":1}
W20260926 11:30:16.133313 7728 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33195 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
I20260926 11:30:16.134227 7797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:16.133778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.98.1:58923)
0926 11:30:16.134051 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:16.134131 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":114,"threads_started":1}
I20260926 11:30:16.135010 7798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:16.134608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.98.2:54057)
0926 11:30:16.134814 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:16.134903 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":71,"threads_started":1}
W20260926 11:30:16.135257 7455 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33195 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
W20260926 11:30:16.135370 7592 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33195 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
I20260926 11:30:16.645690 7799 raft_consensus.cc:482] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f281a9b384774f93b3ca169c35839d66)
I20260926 11:30:16.645931 7799 raft_consensus.cc:504] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: false } }
I20260926 11:30:16.646142 7800 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f281a9b384774f93b3ca169c35839d66)
I20260926 11:30:16.646337 7800 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: false } }
I20260926 11:30:16.646996 7799 leader_election.cc:288] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f281a9b384774f93b3ca169c35839d66 (127.0.98.62:33195), b5d89fa8510448b9950d2ef424068e4d (127.0.98.60:39305)
I20260926 11:30:16.647454 7800 leader_election.cc:288] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f281a9b384774f93b3ca169c35839d66 (127.0.98.62:33195), bba2460c1d254f99900a6f71741eae0a (127.0.98.61:41339)
I20260926 11:30:16.647750 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:16.647291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.0.1:60192)
0926 11:30:16.647555 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:16.647630 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":54,"threads_started":1}
I20260926 11:30:16.647825 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:16.647145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.0.1:60186)
0926 11:30:16.647649 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:16.647708 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":63,"threads_started":1}
W20260926 11:30:16.647980 7076 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
W20260926 11:30:16.647975 7259 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
I20260926 11:30:16.648890 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:16.648653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.0.1:60206)
0926 11:30:16.648787 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:16.648833 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 11:30:16.649029 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:16.648689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.0.1:60216)
0926 11:30:16.648901 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:16.648955 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
W20260926 11:30:16.649015 7076 leader_election.cc:334] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f281a9b384774f93b3ca169c35839d66 (127.0.98.62:33195): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
W20260926 11:30:16.649127 7259 leader_election.cc:334] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f281a9b384774f93b3ca169c35839d66 (127.0.98.62:33195): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
I20260926 11:30:16.651413 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:16.647311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41339 (local address 127.0.0.1:59198)
0926 11:30:16.647876 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:16.647890 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:16.647998 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:16.648440 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:16.648444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:16.648463 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:30:16.648580 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:16.648590 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:16.649576 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:16.649583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:16.650374 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:16.650382 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:16.650981 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:16.650984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:16.651121 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:16.651139 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 11:30:16.651224 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 11:30:16.651288 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":44,"thread_start_us":60,"threads_started":1}
I20260926 11:30:16.651437 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:16.647480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39305 (local address 127.0.0.1:46658)
0926 11:30:16.647714 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:16.647755 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:16.647861 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:16.648399 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:16.648403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:16.648423 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:30:16.648537 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:16.648544 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:16.649670 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:16.649673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:16.650432 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:16.650439 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:16.651044 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:16.651047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:16.651136 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:16.651163 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 11:30:16.651245 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 11:30:16.651311 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":50,"threads_started":1}
I20260926 11:30:16.651444 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:16.647412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59198 (local address 127.0.98.61:41339)
0926 11:30:16.648181 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:16.648187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:16.648201 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:30:16.648248 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:16.648250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:16.648309 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:16.648684 (+ 375us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:30:16.648774 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:16.649468 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:16.650487 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:16.650870 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:16.650981 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:16.651035 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:16.651238 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 11:30:16.651327 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":66,"threads_started":1}
I20260926 11:30:16.651441 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:16.647757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46658 (local address 127.0.98.60:39305)
0926 11:30:16.648139 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:16.648143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:16.648158 (+ 15us) server_negotiation.cc:407] Connection header received
0926 11:30:16.648208 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:16.648214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:16.648261 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:16.648646 (+ 385us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:30:16.648736 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:16.649549 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:16.650546 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:16.650933 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:16.651042 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:16.651115 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:16.651258 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 11:30:16.651333 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":321,"thread_start_us":60,"threads_started":1}
I20260926 11:30:16.652037 7094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b5d89fa8510448b9950d2ef424068e4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bba2460c1d254f99900a6f71741eae0a" is_pre_election: true
I20260926 11:30:16.652043 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bba2460c1d254f99900a6f71741eae0a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b5d89fa8510448b9950d2ef424068e4d" is_pre_election: true
I20260926 11:30:16.652213 7284 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bba2460c1d254f99900a6f71741eae0a in term 1.
I20260926 11:30:16.652213 7094 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5d89fa8510448b9950d2ef424068e4d in term 1.
I20260926 11:30:16.652544 7076 leader_election.cc:302] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b5d89fa8510448b9950d2ef424068e4d, bba2460c1d254f99900a6f71741eae0a; no voters: f281a9b384774f93b3ca169c35839d66
I20260926 11:30:16.652608 7262 leader_election.cc:302] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b5d89fa8510448b9950d2ef424068e4d, bba2460c1d254f99900a6f71741eae0a; no voters: f281a9b384774f93b3ca169c35839d66
I20260926 11:30:16.652762 7799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 11:30:16.652820 7799 raft_consensus.cc:482] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [term 1 FOLLOWER]: Starting leader election (detected failure of leader f281a9b384774f93b3ca169c35839d66)
I20260926 11:30:16.652858 7799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [term 1 FOLLOWER]: Advancing to term 2
I20260926 11:30:16.652956 7800 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 11:30:16.653007 7800 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [term 1 FOLLOWER]: Starting leader election (detected failure of leader f281a9b384774f93b3ca169c35839d66)
I20260926 11:30:16.653064 7800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [term 1 FOLLOWER]: Advancing to term 2
I20260926 11:30:16.654106 7799 raft_consensus.cc:504] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: false } }
I20260926 11:30:16.654106 7800 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: false } }
I20260926 11:30:16.654489 7800 leader_election.cc:288] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [CANDIDATE]: Term 2 election: Requested vote from peers f281a9b384774f93b3ca169c35839d66 (127.0.98.62:33195), bba2460c1d254f99900a6f71741eae0a (127.0.98.61:41339)
I20260926 11:30:16.654481 7799 leader_election.cc:288] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [CANDIDATE]: Term 2 election: Requested vote from peers f281a9b384774f93b3ca169c35839d66 (127.0.98.62:33195), b5d89fa8510448b9950d2ef424068e4d (127.0.98.60:39305)
I20260926 11:30:16.654781 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:16.654586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.0.1:60228)
0926 11:30:16.654675 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:16.654723 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 11:30:16.654866 7094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b5d89fa8510448b9950d2ef424068e4d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bba2460c1d254f99900a6f71741eae0a"
I20260926 11:30:16.654920 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bba2460c1d254f99900a6f71741eae0a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b5d89fa8510448b9950d2ef424068e4d"
I20260926 11:30:16.655059 7284 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate bba2460c1d254f99900a6f71741eae0a in current term 2: Already voted for candidate b5d89fa8510448b9950d2ef424068e4d in this term.
I20260926 11:30:16.655059 7094 raft_consensus.cc:2379] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b5d89fa8510448b9950d2ef424068e4d in current term 2: Already voted for candidate bba2460c1d254f99900a6f71741eae0a in this term.
I20260926 11:30:16.655164 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:16.654586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.0.1:60226)
0926 11:30:16.655030 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:16.655102 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339}
I20260926 11:30:16.655673 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:16.655488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.0.1:60248)
0926 11:30:16.655577 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:16.655622 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 11:30:16.655840 7076 leader_election.cc:334] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f281a9b384774f93b3ca169c35839d66 (127.0.98.62:33195): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
I20260926 11:30:16.655932 7076 leader_election.cc:302] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bba2460c1d254f99900a6f71741eae0a; no voters: b5d89fa8510448b9950d2ef424068e4d, f281a9b384774f93b3ca169c35839d66
I20260926 11:30:16.656059 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:16.655800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.0.1:60252)
0926 11:30:16.655911 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:16.655997 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 11:30:16.656180 7799 raft_consensus.cc:2726] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260926 11:30:16.656216 7259 leader_election.cc:334] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f281a9b384774f93b3ca169c35839d66 (127.0.98.62:33195): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
I20260926 11:30:16.656275 7259 leader_election.cc:302] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b5d89fa8510448b9950d2ef424068e4d; no voters: bba2460c1d254f99900a6f71741eae0a, f281a9b384774f93b3ca169c35839d66
I20260926 11:30:16.656401 7800 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 11:30:17.134410 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:17.133935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.98.3:36335)
0926 11:30:17.134206 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:17.134271 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":112,"threads_started":1}
I20260926 11:30:17.136469 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:17.136047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.98.2:47995)
0926 11:30:17.136304 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:17.136369 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":93,"threads_started":1}
I20260926 11:30:17.136577 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:17.135911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.98.1:49597)
0926 11:30:17.136396 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:17.136478 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":101,"threads_started":1}
I20260926 11:30:18.135650 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.135156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.98.3:37427)
0926 11:30:18.135455 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:18.135528 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":107,"threads_started":1}
I20260926 11:30:18.137672 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.137206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.98.1:53463)
0926 11:30:18.137487 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:18.137554 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":76,"threads_started":1}
I20260926 11:30:18.137728 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.137179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.98.2:47803)
0926 11:30:18.137553 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:18.137623 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":75,"threads_started":1}
I20260926 11:30:18.321394 7815 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:30:18.321564 7815 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: false } }
I20260926 11:30:18.322047 7815 leader_election.cc:288] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f281a9b384774f93b3ca169c35839d66 (127.0.98.62:33195), bba2460c1d254f99900a6f71741eae0a (127.0.98.61:41339)
I20260926 11:30:18.322530 7094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b5d89fa8510448b9950d2ef424068e4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bba2460c1d254f99900a6f71741eae0a" is_pre_election: true
I20260926 11:30:18.322661 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.322282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.0.1:60504)
0926 11:30:18.322513 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:18.322576 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":82,"threads_started":1}
I20260926 11:30:18.322697 7094 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5d89fa8510448b9950d2ef424068e4d in term 2.
I20260926 11:30:18.323040 7262 leader_election.cc:302] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b5d89fa8510448b9950d2ef424068e4d, bba2460c1d254f99900a6f71741eae0a; no voters:
I20260926 11:30:18.323230 7815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260926 11:30:18.323293 7815 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:30:18.323390 7815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [term 2 FOLLOWER]: Advancing to term 3
I20260926 11:30:18.323537 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.323269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.0.1:60510)
0926 11:30:18.323408 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:18.323471 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260926 11:30:18.323647 7259 leader_election.cc:334] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f281a9b384774f93b3ca169c35839d66 (127.0.98.62:33195): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
I20260926 11:30:18.324525 7815 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: false } }
I20260926 11:30:18.324882 7815 leader_election.cc:288] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [CANDIDATE]: Term 3 election: Requested vote from peers f281a9b384774f93b3ca169c35839d66 (127.0.98.62:33195), bba2460c1d254f99900a6f71741eae0a (127.0.98.61:41339)
I20260926 11:30:18.325251 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.325016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.0.1:60518)
0926 11:30:18.325127 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:18.325181 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 11:30:18.325268 7094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b5d89fa8510448b9950d2ef424068e4d" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bba2460c1d254f99900a6f71741eae0a"
I20260926 11:30:18.325434 7094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [term 2 FOLLOWER]: Advancing to term 3
I20260926 11:30:18.325902 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.325748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.0.1:60520)
0926 11:30:18.325820 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:18.325857 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260926 11:30:18.326057 7259 leader_election.cc:334] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer f281a9b384774f93b3ca169c35839d66 (127.0.98.62:33195): Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
I20260926 11:30:18.326669 7094 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5d89fa8510448b9950d2ef424068e4d in term 3.
I20260926 11:30:18.326970 7262 leader_election.cc:302] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b5d89fa8510448b9950d2ef424068e4d, bba2460c1d254f99900a6f71741eae0a; no voters: f281a9b384774f93b3ca169c35839d66
I20260926 11:30:18.327180 7815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [term 3 FOLLOWER]: Leader election won for term 3
I20260926 11:30:18.327466 7815 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [term 3 LEADER]: Becoming Leader. State: Replica: b5d89fa8510448b9950d2ef424068e4d, State: Running, Role: LEADER
I20260926 11:30:18.327793 7815 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: false } }
I20260926 11:30:18.329092 7818 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5d89fa8510448b9950d2ef424068e4d. Latest consensus state: current_term: 3 leader_uuid: "b5d89fa8510448b9950d2ef424068e4d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: false } } }
I20260926 11:30:18.329217 7818 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:18.329497 7820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:30:18.331643 7820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:30:18.332072 7820 catalog_manager.cc:1259] Loaded cluster ID: 7d38a3fb7f064528b10e3eedc9ccf1ce
I20260926 11:30:18.332124 7820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:30:18.332647 7820 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:30:18.332988 7820 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d: Loaded TSK: 0
I20260926 11:30:18.333624 7820 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:30:18.393784 7275 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46644:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:30:18.397018 7275 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:30:18.400230 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.399960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.0.1:60538)
0926 11:30:18.400081 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:18.400147 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 11:30:18.400362 7094 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [term 3 FOLLOWER]: Refusing update from remote peer b5d89fa8510448b9950d2ef424068e4d: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260926 11:30:18.400913 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.400667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.0.1:60554)
0926 11:30:18.400777 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:18.400826 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 11:30:18.400840 7818 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260926 11:30:18.401037 7259 consensus_peers.cc:487] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d -> Peer f281a9b384774f93b3ca169c35839d66 (127.0.98.62:33195): Couldn't send request to peer f281a9b384774f93b3ca169c35839d66. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 11:30:18.402346 7822 sys_catalog.cc:455] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5d89fa8510448b9950d2ef424068e4d. Latest consensus state: current_term: 3 leader_uuid: "b5d89fa8510448b9950d2ef424068e4d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: false } } }
I20260926 11:30:18.402493 7822 sys_catalog.cc:458] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:30:18.403616 7815 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "b5d89fa8510448b9950d2ef424068e4d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: false } } }
I20260926 11:30:18.403762 7815 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:18.404037 7822 sys_catalog.cc:455] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "b5d89fa8510448b9950d2ef424068e4d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "f281a9b384774f93b3ca169c35839d66" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33195 } } peers { permanent_uuid: "bba2460c1d254f99900a6f71741eae0a" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41339 } attrs { promote: false } } peers { permanent_uuid: "b5d89fa8510448b9950d2ef424068e4d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39305 } attrs { promote: false } } }
I20260926 11:30:18.404157 7822 sys_catalog.cc:458] T 00000000000000000000000000000000 P bba2460c1d254f99900a6f71741eae0a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:30:18.414667 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.410931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33814 (local address 127.0.98.3:46037)
0926 11:30:18.411164 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:18.411167 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:18.411178 (+ 11us) server_negotiation.cc:407] Connection header received
0926 11:30:18.411218 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:18.411220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:18.411289 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:18.411406 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:30:18.411752 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:18.412270 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:18.413692 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:18.414160 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:18.414291 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:18.414357 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:18.414470 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 11:30:18.414536 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":65,"threads_started":1}
I20260926 11:30:18.414667 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.409902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46037 (local address 127.0.0.1:33814)
0926 11:30:18.410016 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:18.410039 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:18.410143 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:18.411416 (+ 1273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:18.411419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:18.411447 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:30:18.411610 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:18.411616 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:18.412389 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:18.412392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:18.413564 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:18.413573 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:18.414280 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:18.414284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:18.414363 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:18.414391 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 11:30:18.414459 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 11:30:18.414526 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":36,"spinlock_wait_cycles":5248}
I20260926 11:30:18.415022 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.410335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42249 (local address 127.0.0.1:43590)
0926 11:30:18.410581 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:18.410596 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:18.410674 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:18.411416 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:18.411419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:18.411448 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:30:18.411621 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:18.411627 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:18.412629 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:18.412632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:18.413726 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:18.413733 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:18.414589 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:18.414593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:18.414694 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:18.414739 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 11:30:18.414789 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 11:30:18.414872 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":51,"spinlock_wait_cycles":6144,"thread_start_us":112,"threads_started":1}
I20260926 11:30:18.415295 7661 tablet_service.cc:1467] Processing CreateTablet for tablet 35c6ca4ba3c84e64b7fe83f8fd3e432b (DEFAULT_TABLE table=test-table [id=cb31028b37fd49adbfeb106bfbc5df17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:30:18.415446 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.410433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43590 (local address 127.0.98.2:42249)
0926 11:30:18.411172 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:18.411177 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:18.411189 (+ 12us) server_negotiation.cc:407] Connection header received
0926 11:30:18.411254 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:18.411256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:18.411300 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:18.411496 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:30:18.411770 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:18.412524 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:18.413853 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:18.414466 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:18.415103 (+ 637us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:18.415176 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:18.415215 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 11:30:18.415305 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":79,"threads_started":1}
I20260926 11:30:18.415483 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.410782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44862 (local address 127.0.98.1:46203)
0926 11:30:18.411044 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:18.411050 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:18.411067 (+ 17us) server_negotiation.cc:407] Connection header received
0926 11:30:18.411121 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:18.411125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:18.411171 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:18.411264 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:30:18.411557 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:18.412403 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:18.413804 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:18.414418 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:18.414531 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:18.414605 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:18.415278 (+ 673us) server_negotiation.cc:299] Negotiation successful
0926 11:30:18.415361 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":83,"threads_started":1}
I20260926 11:30:18.415023 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.410663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46203 (local address 127.0.0.1:44862)
0926 11:30:18.410886 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:18.410900 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:18.410980 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:18.411284 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:18.411287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:18.411305 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:30:18.411432 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:18.411438 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:18.412526 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:18.412673 (+ 147us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:18.413685 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:18.413694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:18.414651 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:18.414653 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:18.414778 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:18.414801 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 11:30:18.414857 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 11:30:18.414907 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":36,"thread_start_us":87,"threads_started":1}
I20260926 11:30:18.415637 7661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35c6ca4ba3c84e64b7fe83f8fd3e432b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:18.416272 7388 tablet_service.cc:1467] Processing CreateTablet for tablet 35c6ca4ba3c84e64b7fe83f8fd3e432b (DEFAULT_TABLE table=test-table [id=cb31028b37fd49adbfeb106bfbc5df17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:30:18.416513 7388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35c6ca4ba3c84e64b7fe83f8fd3e432b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:18.416500 7525 tablet_service.cc:1467] Processing CreateTablet for tablet 35c6ca4ba3c84e64b7fe83f8fd3e432b (DEFAULT_TABLE table=test-table [id=cb31028b37fd49adbfeb106bfbc5df17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:30:18.416774 7525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35c6ca4ba3c84e64b7fe83f8fd3e432b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:18.418375 7832 tablet_bootstrap.cc:492] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 5fb677408a224fa2b4a8eee4dc918823: Bootstrap starting.
I20260926 11:30:18.419029 7833 tablet_bootstrap.cc:492] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 26d2d13738244b0399f95e5e8a836d82: Bootstrap starting.
I20260926 11:30:18.419682 7832 tablet_bootstrap.cc:654] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 5fb677408a224fa2b4a8eee4dc918823: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:18.419952 7833 tablet_bootstrap.cc:654] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 26d2d13738244b0399f95e5e8a836d82: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:18.419945 7834 tablet_bootstrap.cc:492] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 4b3cc5c906eb42cca07bc2854fc9ff3c: Bootstrap starting.
I20260926 11:30:18.420819 7834 tablet_bootstrap.cc:654] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 4b3cc5c906eb42cca07bc2854fc9ff3c: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:18.420993 7832 tablet_bootstrap.cc:492] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 5fb677408a224fa2b4a8eee4dc918823: No bootstrap required, opened a new log
I20260926 11:30:18.421079 7832 ts_tablet_manager.cc:1397] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 5fb677408a224fa2b4a8eee4dc918823: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 11:30:18.421497 7832 raft_consensus.cc:348] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 5fb677408a224fa2b4a8eee4dc918823 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fb677408a224fa2b4a8eee4dc918823" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46037 } } peers { permanent_uuid: "26d2d13738244b0399f95e5e8a836d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42249 } } peers { permanent_uuid: "4b3cc5c906eb42cca07bc2854fc9ff3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46203 } }
I20260926 11:30:18.421633 7832 raft_consensus.cc:374] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 5fb677408a224fa2b4a8eee4dc918823 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:18.421665 7832 raft_consensus.cc:729] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 5fb677408a224fa2b4a8eee4dc918823 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fb677408a224fa2b4a8eee4dc918823, State: Initialized, Role: FOLLOWER
I20260926 11:30:18.421797 7832 consensus_queue.cc:260] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 5fb677408a224fa2b4a8eee4dc918823 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fb677408a224fa2b4a8eee4dc918823" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46037 } } peers { permanent_uuid: "26d2d13738244b0399f95e5e8a836d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42249 } } peers { permanent_uuid: "4b3cc5c906eb42cca07bc2854fc9ff3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46203 } }
I20260926 11:30:18.422001 7834 tablet_bootstrap.cc:492] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 4b3cc5c906eb42cca07bc2854fc9ff3c: No bootstrap required, opened a new log
I20260926 11:30:18.422084 7834 ts_tablet_manager.cc:1397] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 4b3cc5c906eb42cca07bc2854fc9ff3c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 11:30:18.422173 7832 ts_tablet_manager.cc:1428] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 5fb677408a224fa2b4a8eee4dc918823: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 11:30:18.422466 7834 raft_consensus.cc:348] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 4b3cc5c906eb42cca07bc2854fc9ff3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fb677408a224fa2b4a8eee4dc918823" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46037 } } peers { permanent_uuid: "26d2d13738244b0399f95e5e8a836d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42249 } } peers { permanent_uuid: "4b3cc5c906eb42cca07bc2854fc9ff3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46203 } }
I20260926 11:30:18.422575 7834 raft_consensus.cc:374] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 4b3cc5c906eb42cca07bc2854fc9ff3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:18.422608 7834 raft_consensus.cc:729] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 4b3cc5c906eb42cca07bc2854fc9ff3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b3cc5c906eb42cca07bc2854fc9ff3c, State: Initialized, Role: FOLLOWER
I20260926 11:30:18.422752 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.422513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.98.3:37967)
0926 11:30:18.422614 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:18.422675 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 11:30:18.422726 7834 consensus_queue.cc:260] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 4b3cc5c906eb42cca07bc2854fc9ff3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fb677408a224fa2b4a8eee4dc918823" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46037 } } peers { permanent_uuid: "26d2d13738244b0399f95e5e8a836d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42249 } } peers { permanent_uuid: "4b3cc5c906eb42cca07bc2854fc9ff3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46203 } }
I20260926 11:30:18.423240 7834 ts_tablet_manager.cc:1428] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 4b3cc5c906eb42cca07bc2854fc9ff3c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 11:30:18.424265 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.423997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.98.1:36691)
0926 11:30:18.424145 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:18.424199 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 11:30:18.424306 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.424068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.98.3:46701)
0926 11:30:18.424199 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:18.424246 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 11:30:18.425410 7727 heartbeater.cc:503] Master 127.0.98.60:39305 was elected leader, sending a full tablet report...
I20260926 11:30:18.425436 7454 heartbeater.cc:503] Master 127.0.98.60:39305 was elected leader, sending a full tablet report...
I20260926 11:30:18.425948 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.425621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.98.1:55981)
0926 11:30:18.425829 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:18.425878 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260926 11:30:18.425997 7833 tablet_bootstrap.cc:492] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 26d2d13738244b0399f95e5e8a836d82: No bootstrap required, opened a new log
I20260926 11:30:18.426086 7833 ts_tablet_manager.cc:1397] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 26d2d13738244b0399f95e5e8a836d82: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260926 11:30:18.426625 7833 raft_consensus.cc:348] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 26d2d13738244b0399f95e5e8a836d82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fb677408a224fa2b4a8eee4dc918823" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46037 } } peers { permanent_uuid: "26d2d13738244b0399f95e5e8a836d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42249 } } peers { permanent_uuid: "4b3cc5c906eb42cca07bc2854fc9ff3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46203 } }
I20260926 11:30:18.426746 7833 raft_consensus.cc:374] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 26d2d13738244b0399f95e5e8a836d82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:18.426783 7833 raft_consensus.cc:729] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 26d2d13738244b0399f95e5e8a836d82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d2d13738244b0399f95e5e8a836d82, State: Initialized, Role: FOLLOWER
I20260926 11:30:18.427616 7833 consensus_queue.cc:260] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 26d2d13738244b0399f95e5e8a836d82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fb677408a224fa2b4a8eee4dc918823" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46037 } } peers { permanent_uuid: "26d2d13738244b0399f95e5e8a836d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42249 } } peers { permanent_uuid: "4b3cc5c906eb42cca07bc2854fc9ff3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46203 } }
I20260926 11:30:18.428077 7833 ts_tablet_manager.cc:1428] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 26d2d13738244b0399f95e5e8a836d82: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260926 11:30:18.428656 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.428319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.98.2:41233)
0926 11:30:18.428519 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:18.428588 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260926 11:30:18.429294 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.429081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.98.2:46607)
0926 11:30:18.429184 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:18.429236 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 11:30:18.429385 7591 heartbeater.cc:503] Master 127.0.98.60:39305 was elected leader, sending a full tablet report...
I20260926 11:30:18.689812 7838 raft_consensus.cc:482] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 5fb677408a224fa2b4a8eee4dc918823 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:30:18.690002 7838 raft_consensus.cc:504] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 5fb677408a224fa2b4a8eee4dc918823 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fb677408a224fa2b4a8eee4dc918823" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46037 } } peers { permanent_uuid: "26d2d13738244b0399f95e5e8a836d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42249 } } peers { permanent_uuid: "4b3cc5c906eb42cca07bc2854fc9ff3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46203 } }
I20260926 11:30:18.691074 7838 leader_election.cc:288] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 5fb677408a224fa2b4a8eee4dc918823 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26d2d13738244b0399f95e5e8a836d82 (127.0.98.2:42249), 4b3cc5c906eb42cca07bc2854fc9ff3c (127.0.98.1:46203)
I20260926 11:30:18.696118 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.691818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38537 (local address 127.0.98.1:46203)
0926 11:30:18.691966 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:18.691970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:18.692117 (+ 147us) server_negotiation.cc:407] Connection header received
0926 11:30:18.692226 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:18.692228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:18.692290 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:18.692376 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:30:18.692585 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:18.693282 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:18.694833 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:18.695483 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:18.695572 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:18.695633 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:18.695926 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 11:30:18.695992 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 11:30:18.696118 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.691477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46699 (local address 127.0.98.2:42249)
0926 11:30:18.691617 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:18.691621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:18.691637 (+ 16us) server_negotiation.cc:407] Connection header received
0926 11:30:18.691787 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:18.691790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:18.691868 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:18.691945 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:30:18.692326 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:18.693134 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:18.694833 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:18.695438 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:18.695556 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:18.695633 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:18.695926 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 11:30:18.695992 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 11:30:18.696159 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.691361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42249 (local address 127.0.98.3:46699)
0926 11:30:18.691485 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:18.691499 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:18.691635 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:18.692028 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:18.692032 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:18.692062 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:30:18.692214 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:18.692223 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:18.693259 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:18.693262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:18.694713 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:18.694718 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:18.695597 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:18.695601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:18.695773 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:18.695827 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 11:30:18.695896 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 11:30:18.696004 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":129}
I20260926 11:30:18.696159 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.691679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46203 (local address 127.0.98.3:38537)
0926 11:30:18.692026 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:18.692039 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:18.692131 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:18.692392 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:18.692393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:18.692409 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:30:18.692487 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:18.692491 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:18.693377 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:18.693380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:18.694707 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:18.694717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:18.695597 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:18.695601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:18.695779 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:18.695817 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 11:30:18.695896 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 11:30:18.696004 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":97,"thread_start_us":173,"threads_started":1}
I20260926 11:30:18.696611 7545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35c6ca4ba3c84e64b7fe83f8fd3e432b" candidate_uuid: "5fb677408a224fa2b4a8eee4dc918823" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d2d13738244b0399f95e5e8a836d82" is_pre_election: true
I20260926 11:30:18.696801 7545 raft_consensus.cc:2454] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 26d2d13738244b0399f95e5e8a836d82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fb677408a224fa2b4a8eee4dc918823 in term 0.
I20260926 11:30:18.697182 7614 leader_election.cc:302] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 5fb677408a224fa2b4a8eee4dc918823 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d2d13738244b0399f95e5e8a836d82, 5fb677408a224fa2b4a8eee4dc918823; no voters:
I20260926 11:30:18.697268 7408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35c6ca4ba3c84e64b7fe83f8fd3e432b" candidate_uuid: "5fb677408a224fa2b4a8eee4dc918823" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b3cc5c906eb42cca07bc2854fc9ff3c" is_pre_election: true
I20260926 11:30:18.697445 7838 raft_consensus.cc:2781] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 5fb677408a224fa2b4a8eee4dc918823 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:30:18.697525 7838 raft_consensus.cc:482] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 5fb677408a224fa2b4a8eee4dc918823 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:30:18.697525 7408 raft_consensus.cc:2454] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 4b3cc5c906eb42cca07bc2854fc9ff3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fb677408a224fa2b4a8eee4dc918823 in term 0.
I20260926 11:30:18.697576 7838 raft_consensus.cc:3037] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 5fb677408a224fa2b4a8eee4dc918823 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:18.698601 7838 raft_consensus.cc:504] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 5fb677408a224fa2b4a8eee4dc918823 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fb677408a224fa2b4a8eee4dc918823" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46037 } } peers { permanent_uuid: "26d2d13738244b0399f95e5e8a836d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42249 } } peers { permanent_uuid: "4b3cc5c906eb42cca07bc2854fc9ff3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46203 } }
I20260926 11:30:18.698931 7838 leader_election.cc:288] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 5fb677408a224fa2b4a8eee4dc918823 [CANDIDATE]: Term 1 election: Requested vote from peers 26d2d13738244b0399f95e5e8a836d82 (127.0.98.2:42249), 4b3cc5c906eb42cca07bc2854fc9ff3c (127.0.98.1:46203)
I20260926 11:30:18.699241 7545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35c6ca4ba3c84e64b7fe83f8fd3e432b" candidate_uuid: "5fb677408a224fa2b4a8eee4dc918823" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26d2d13738244b0399f95e5e8a836d82"
I20260926 11:30:18.699386 7545 raft_consensus.cc:3037] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 26d2d13738244b0399f95e5e8a836d82 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:18.699357 7408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35c6ca4ba3c84e64b7fe83f8fd3e432b" candidate_uuid: "5fb677408a224fa2b4a8eee4dc918823" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b3cc5c906eb42cca07bc2854fc9ff3c"
I20260926 11:30:18.699466 7408 raft_consensus.cc:3037] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 4b3cc5c906eb42cca07bc2854fc9ff3c [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:18.700585 7545 raft_consensus.cc:2454] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 26d2d13738244b0399f95e5e8a836d82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fb677408a224fa2b4a8eee4dc918823 in term 1.
I20260926 11:30:18.700605 7408 raft_consensus.cc:2454] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 4b3cc5c906eb42cca07bc2854fc9ff3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fb677408a224fa2b4a8eee4dc918823 in term 1.
I20260926 11:30:18.700929 7614 leader_election.cc:302] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 5fb677408a224fa2b4a8eee4dc918823 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26d2d13738244b0399f95e5e8a836d82, 5fb677408a224fa2b4a8eee4dc918823; no voters:
I20260926 11:30:18.701107 7838 raft_consensus.cc:2781] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 5fb677408a224fa2b4a8eee4dc918823 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:30:18.701382 7838 raft_consensus.cc:686] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 5fb677408a224fa2b4a8eee4dc918823 [term 1 LEADER]: Becoming Leader. State: Replica: 5fb677408a224fa2b4a8eee4dc918823, State: Running, Role: LEADER
I20260926 11:30:18.701632 7838 consensus_queue.cc:237] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 5fb677408a224fa2b4a8eee4dc918823 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fb677408a224fa2b4a8eee4dc918823" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46037 } } peers { permanent_uuid: "26d2d13738244b0399f95e5e8a836d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42249 } } peers { permanent_uuid: "4b3cc5c906eb42cca07bc2854fc9ff3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46203 } }
I20260926 11:30:18.703178 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.702921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.98.3:59519)
0926 11:30:18.703053 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:18.703107 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260926 11:30:18.704078 7275 catalog_manager.cc:5495] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 5fb677408a224fa2b4a8eee4dc918823 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fb677408a224fa2b4a8eee4dc918823 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5fb677408a224fa2b4a8eee4dc918823" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5fb677408a224fa2b4a8eee4dc918823" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26d2d13738244b0399f95e5e8a836d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b3cc5c906eb42cca07bc2854fc9ff3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46203 } health_report { overall_health: UNKNOWN } } }
I20260926 11:30:18.780305 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.776729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33828 (local address 127.0.98.3:46037)
0926 11:30:18.776896 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:18.776902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:18.776919 (+ 17us) server_negotiation.cc:407] Connection header received
0926 11:30:18.777001 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:18.777003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:18.777052 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:18.777165 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:30:18.777415 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:18.778007 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:18.778842 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:18.779072 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:18.779185 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:18.779457 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:30:18.779927 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:30:18.780058 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:18.780099 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 11:30:18.780154 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 11:30:18.783483 7545 raft_consensus.cc:1261] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 26d2d13738244b0399f95e5e8a836d82 [term 1 FOLLOWER]: Refusing update from remote peer 5fb677408a224fa2b4a8eee4dc918823: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:30:18.783767 7408 raft_consensus.cc:1261] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 4b3cc5c906eb42cca07bc2854fc9ff3c [term 1 FOLLOWER]: Refusing update from remote peer 5fb677408a224fa2b4a8eee4dc918823: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:30:18.784111 7842 consensus_queue.cc:1035] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 5fb677408a224fa2b4a8eee4dc918823 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26d2d13738244b0399f95e5e8a836d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:30:18.784366 7838 consensus_queue.cc:1035] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 5fb677408a224fa2b4a8eee4dc918823 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b3cc5c906eb42cca07bc2854fc9ff3c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:30:18.786022 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.785733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.98.2:54453)
0926 11:30:18.785880 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:18.785937 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 11:30:18.786441 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.786138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.98.1:56317)
0926 11:30:18.786325 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:18.786376 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260926 11:30:18.790673 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.790011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.98.2:49627)
0926 11:30:18.790535 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:18.790595 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458}
I20260926 11:30:18.790951 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.790045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.98.1:52181)
0926 11:30:18.790833 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:18.790888 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":718}
I20260926 11:30:18.792086 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.791792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.98.3:32915)
0926 11:30:18.791963 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:18.792018 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 11:30:18.797714 7275 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:30:18.797881 7275 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46644:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:30:18.801614 7275 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5d89fa8510448b9950d2ef424068e4d: Sending DeleteTablet for 3 replicas of tablet 35c6ca4ba3c84e64b7fe83f8fd3e432b
I20260926 11:30:18.802485 7388 tablet_service.cc:1514] Processing DeleteTablet for tablet 35c6ca4ba3c84e64b7fe83f8fd3e432b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:30:18 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:44862
I20260926 11:30:18.802651 7525 tablet_service.cc:1514] Processing DeleteTablet for tablet 35c6ca4ba3c84e64b7fe83f8fd3e432b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:30:18 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:43590
I20260926 11:30:18.802663 7661 tablet_service.cc:1514] Processing DeleteTablet for tablet 35c6ca4ba3c84e64b7fe83f8fd3e432b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:30:18 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33814
I20260926 11:30:18.802858 7856 tablet_replica.cc:330] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 4b3cc5c906eb42cca07bc2854fc9ff3c: stopping tablet replica
I20260926 11:30:18.802980 7856 raft_consensus.cc:2229] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 4b3cc5c906eb42cca07bc2854fc9ff3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:30:18.803107 7856 raft_consensus.cc:2258] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 4b3cc5c906eb42cca07bc2854fc9ff3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:30:18.803426 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 7236
I20260926 11:30:18.807986 7858 tablet_replica.cc:330] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 5fb677408a224fa2b4a8eee4dc918823: stopping tablet replica
I20260926 11:30:18.808111 7858 raft_consensus.cc:2229] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 5fb677408a224fa2b4a8eee4dc918823 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:30:18.808404 7858 raft_consensus.cc:2258] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 5fb677408a224fa2b4a8eee4dc918823 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:30:18.808631 7857 tablet_replica.cc:330] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 26d2d13738244b0399f95e5e8a836d82: stopping tablet replica
I20260926 11:30:18.808756 7857 raft_consensus.cc:2229] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 26d2d13738244b0399f95e5e8a836d82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:30:18.808887 7857 raft_consensus.cc:2258] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 26d2d13738244b0399f95e5e8a836d82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:30:18.809283 7858 ts_tablet_manager.cc:1905] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 5fb677408a224fa2b4a8eee4dc918823: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:30:18.809563 7857 ts_tablet_manager.cc:1905] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 26d2d13738244b0399f95e5e8a836d82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:30:18.812045 7857 ts_tablet_manager.cc:1918] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 26d2d13738244b0399f95e5e8a836d82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:30:18.812125 7857 log.cc:1196] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 26d2d13738244b0399f95e5e8a836d82: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-1/wal/wals/35c6ca4ba3c84e64b7fe83f8fd3e432b
I20260926 11:30:18.812412 7857 ts_tablet_manager.cc:1939] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 26d2d13738244b0399f95e5e8a836d82: Deleting consensus metadata
I20260926 11:30:18.812744 7858 ts_tablet_manager.cc:1918] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 5fb677408a224fa2b4a8eee4dc918823: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:30:18.812879 7858 log.cc:1196] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 5fb677408a224fa2b4a8eee4dc918823: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790422180210734-392-0/minicluster-data/ts-2/wal/wals/35c6ca4ba3c84e64b7fe83f8fd3e432b
I20260926 11:30:18.813133 7259 catalog_manager.cc:4841] TS 26d2d13738244b0399f95e5e8a836d82 (127.0.98.2:42249): tablet 35c6ca4ba3c84e64b7fe83f8fd3e432b (table test-table [id=cb31028b37fd49adbfeb106bfbc5df17]) successfully deleted
I20260926 11:30:18.813241 7858 ts_tablet_manager.cc:1939] T 35c6ca4ba3c84e64b7fe83f8fd3e432b P 5fb677408a224fa2b4a8eee4dc918823: Deleting consensus metadata
W20260926 11:30:18.813836 7829 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:18.812803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46203 (local address 127.0.0.1:44876)
0926 11:30:18.812919 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:18.812938 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:18.813033 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:18.813727 (+ 694us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46203: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 11:30:18.813881 7262 catalog_manager.cc:4841] TS 5fb677408a224fa2b4a8eee4dc918823 (127.0.98.3:46037): tablet 35c6ca4ba3c84e64b7fe83f8fd3e432b (table test-table [id=cb31028b37fd49adbfeb106bfbc5df17]) successfully deleted
W20260926 11:30:18.813992 7259 catalog_manager.cc:4584] TS 4b3cc5c906eb42cca07bc2854fc9ff3c (127.0.98.1:46203): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 35c6ca4ba3c84e64b7fe83f8fd3e432b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46203: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 11:30:18.814256 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 7462
I20260926 11:30:18.818353 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.817893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.0.1:60562)
0926 11:30:18.818103 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:18.818193 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260926 11:30:18.819384 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:18.819118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33195 (local address 127.0.0.1:60574)
0926 11:30:18.819282 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:18.819326 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 11:30:18.821882 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 7598
I20260926 11:30:18.828073 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 6956
I20260926 11:30:18.833748 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 7064
2026-09-26T11:30:18Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7338 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:30:18 dist-test-slave-mrck krb5kdc[7861](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:30:18 dist-test-slave-mrck krb5kdc[7861](info): set up 2 sockets
Sep 26 11:30:18 dist-test-slave-mrck krb5kdc[7861](info): commencing operation
krb5kdc: starting...
W20260926 11:30:19.900398 392 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:30:19 dist-test-slave-mrck krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:30:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:30:19Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260926 11:30:20.018987 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36439
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:20.129272 7874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:20.129546 7874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:20.129587 7874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:20.133539 7874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:30:20.133596 7874 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 11:30:20.133615 7874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:20.133633 7874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:20.133648 7874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:30:20.133663 7874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:30:20.138855 7874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36439
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:20.139874 7874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:20.140825 7874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 11:30:20.149497 7874 server_base.cc:1034] running on GCE node
W20260926 11:30:20.149557 7882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:20.149569 7880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:20.149673 7879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:20.150359 7874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:20.151480 7874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:20.152686 7874 hybrid_clock.cc:648] HybridClock initialized: now 1790422220152655 us; error 47 us; skew 500 ppm
Sep 26 11:30:20 dist-test-slave-mrck krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422220, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:20.155419 7874 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260926 11:30:20.156405 7874 webserver.cc:460] Webserver started at http://127.0.98.62:37049/ using document root <none> and password file <none>
I20260926 11:30:20.156970 7874 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:20.157055 7874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:20.157284 7874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:20.159284 7874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "6c53348dae04456b9c59629c03d412d6"
format_stamp: "Formatted at 2026-09-26 11:30:20 on dist-test-slave-mrck"
server_key: "9a49a394c10a0e5265f6a1a9e3428853"
server_key_iv: "7bdda9d0f932e6ec5a8561d0c269d92c"
server_key_version: "encryptionkey@0"
I20260926 11:30:20.159909 7874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "6c53348dae04456b9c59629c03d412d6"
format_stamp: "Formatted at 2026-09-26 11:30:20 on dist-test-slave-mrck"
server_key: "9a49a394c10a0e5265f6a1a9e3428853"
server_key_iv: "7bdda9d0f932e6ec5a8561d0c269d92c"
server_key_version: "encryptionkey@0"
I20260926 11:30:20.163648 7874 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:30:20.166139 7889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:20.167361 7874 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 11:30:20.167497 7874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "6c53348dae04456b9c59629c03d412d6"
format_stamp: "Formatted at 2026-09-26 11:30:20 on dist-test-slave-mrck"
server_key: "9a49a394c10a0e5265f6a1a9e3428853"
server_key_iv: "7bdda9d0f932e6ec5a8561d0c269d92c"
server_key_version: "encryptionkey@0"
I20260926 11:30:20.167611 7874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:20.188776 7874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:20.191805 7874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:20.191998 7874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:20.199807 7874 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36439
I20260926 11:30:20.199834 7951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36439 every 8 connection(s)
I20260926 11:30:20.200732 7874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:30:20.203347 7952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:20.205112 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 7874
I20260926 11:30:20.205251 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:30:20.205505 392 external_mini_cluster.cc:1428] Setting key b06389beeb2024784fdc8b83c968a279
I20260926 11:30:20.209959 7952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c53348dae04456b9c59629c03d412d6: Bootstrap starting.
I20260926 11:30:20.212613 7952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c53348dae04456b9c59629c03d412d6: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:20.213514 7952 log.cc:824] T 00000000000000000000000000000000 P 6c53348dae04456b9c59629c03d412d6: Log is configured to *not* fsync() on all Append() calls
Sep 26 11:30:20 dist-test-slave-mrck krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 11:30:20.215811 7952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c53348dae04456b9c59629c03d412d6: No bootstrap required, opened a new log
I20260926 11:30:20.220119 7952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c53348dae04456b9c59629c03d412d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c53348dae04456b9c59629c03d412d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36439 } }
I20260926 11:30:20.220420 7952 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c53348dae04456b9c59629c03d412d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:20.220486 7952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c53348dae04456b9c59629c03d412d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c53348dae04456b9c59629c03d412d6, State: Initialized, Role: FOLLOWER
I20260926 11:30:20.220961 7952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c53348dae04456b9c59629c03d412d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c53348dae04456b9c59629c03d412d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36439 } }
I20260926 11:30:20.221129 7952 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c53348dae04456b9c59629c03d412d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:30:20.221210 7952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c53348dae04456b9c59629c03d412d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:30:20.221307 7952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c53348dae04456b9c59629c03d412d6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:20.221804 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:20.206863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34260 (local address 127.0.98.62:36439)
0926 11:30:20.207253 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:20.207272 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:20.207314 (+ 42us) server_negotiation.cc:407] Connection header received
0926 11:30:20.208125 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:20.208150 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:20.208574 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:20.208967 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:20.209645 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:20.211069 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:20.212009 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:20.212684 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:20.212874 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:20.215762 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:20.215769 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:20.215795 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:20.215829 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:20.218521 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:20.218988 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:20.218994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:20.219000 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:20.219144 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:20.219428 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:20.219432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:20.219433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:20.219988 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:20.220192 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:20.220416 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 11:30:20.220725 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":105,"threads_started":1}
I20260926 11:30:20.222414 7952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c53348dae04456b9c59629c03d412d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c53348dae04456b9c59629c03d412d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36439 } }
I20260926 11:30:20.222702 7952 leader_election.cc:302] T 00000000000000000000000000000000 P 6c53348dae04456b9c59629c03d412d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c53348dae04456b9c59629c03d412d6; no voters:
I20260926 11:30:20.222980 7952 leader_election.cc:288] T 00000000000000000000000000000000 P 6c53348dae04456b9c59629c03d412d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:30:20.223052 7957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c53348dae04456b9c59629c03d412d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:30:20.223237 7957 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c53348dae04456b9c59629c03d412d6 [term 1 LEADER]: Becoming Leader. State: Replica: 6c53348dae04456b9c59629c03d412d6, State: Running, Role: LEADER
I20260926 11:30:20.223513 7957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c53348dae04456b9c59629c03d412d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c53348dae04456b9c59629c03d412d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36439 } }
I20260926 11:30:20.224264 7952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c53348dae04456b9c59629c03d412d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:30:20.225402 7959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c53348dae04456b9c59629c03d412d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c53348dae04456b9c59629c03d412d6. Latest consensus state: current_term: 1 leader_uuid: "6c53348dae04456b9c59629c03d412d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c53348dae04456b9c59629c03d412d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36439 } } }
I20260926 11:30:20.225409 7958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c53348dae04456b9c59629c03d412d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c53348dae04456b9c59629c03d412d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c53348dae04456b9c59629c03d412d6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36439 } } }
I20260926 11:30:20.225628 7959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c53348dae04456b9c59629c03d412d6 [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:20.225657 7958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c53348dae04456b9c59629c03d412d6 [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:20.226603 7968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:30:20.229621 7968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:30:20.234879 7968 catalog_manager.cc:1347] Generated new cluster ID: 80c363a27f534bf7b2d6a08c09b04d8a
I20260926 11:30:20.234946 7968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:30:20.249861 7968 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:30:20.250777 7968 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:30:20.255908 7968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c53348dae04456b9c59629c03d412d6: Generated new TSK 0
I20260926 11:30:20.256754 7968 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260926 11:30:20.294234 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36439
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:20.439604 7978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:20.439908 7978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:20.439981 7978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:20.443970 7978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:20.444051 7978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:20.444119 7978 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 11:30:20.444193 7978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:30:20.449791 7978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:20.450860 7978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:20.451896 7978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:20.461442 7986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:20.461506 7978 server_base.cc:1034] running on GCE node
W20260926 11:30:20.461436 7983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:20.461434 7984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:20.462059 7978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:20.462621 7978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:20.463831 7978 hybrid_clock.cc:648] HybridClock initialized: now 1790422220463815 us; error 27 us; skew 500 ppm
Sep 26 11:30:20 dist-test-slave-mrck krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422220, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:20.466493 7978 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 11:30:20.467329 7978 webserver.cc:460] Webserver started at http://127.0.98.1:34857/ using document root <none> and password file <none>
I20260926 11:30:20.467947 7978 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:20.468034 7978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:20.468255 7978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:20.470223 7978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-0/data/instance:
uuid: "019b6a6a8c374064be3f9293e1b62077"
format_stamp: "Formatted at 2026-09-26 11:30:20 on dist-test-slave-mrck"
server_key: "d0531affdd09d295859ef8375323c117"
server_key_iv: "17d30c80cfed5ad1df214dd03c86b8f0"
server_key_version: "encryptionkey@0"
I20260926 11:30:20.470762 7978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "019b6a6a8c374064be3f9293e1b62077"
format_stamp: "Formatted at 2026-09-26 11:30:20 on dist-test-slave-mrck"
server_key: "d0531affdd09d295859ef8375323c117"
server_key_iv: "17d30c80cfed5ad1df214dd03c86b8f0"
server_key_version: "encryptionkey@0"
I20260926 11:30:20.474457 7978 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 11:30:20.476894 7993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:20.478052 7978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 11:30:20.478183 7978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "019b6a6a8c374064be3f9293e1b62077"
format_stamp: "Formatted at 2026-09-26 11:30:20 on dist-test-slave-mrck"
server_key: "d0531affdd09d295859ef8375323c117"
server_key_iv: "17d30c80cfed5ad1df214dd03c86b8f0"
server_key_version: "encryptionkey@0"
I20260926 11:30:20.478308 7978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:20.501571 7978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:20.504679 7978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:20.504892 7978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:20.506397 7978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:20.506472 7978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:20.506547 7978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:20.506590 7978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 11:30:20 dist-test-slave-mrck krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422220, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 11:30:20.517717 7978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41563
I20260926 11:30:20.517851 8108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41563 every 8 connection(s)
I20260926 11:30:20.518671 7978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:30:20.520793 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:20.507317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53283 (local address 127.0.98.62:36439)
0926 11:30:20.507437 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:20.507440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:20.508568 (+ 1128us) server_negotiation.cc:407] Connection header received
0926 11:30:20.509757 (+ 1189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:20.509763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:20.509842 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:20.509952 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:20.511494 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:20.512357 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:20.513712 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:20.514078 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:20.514190 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:20.516856 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:20.516859 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:20.516862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:20.516897 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:20.518536 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:20.519078 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:20.519082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:20.519084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:20.519175 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:20.519587 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:20.519591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:20.519593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:20.519870 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:20.519957 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:20.520542 (+ 585us) server_negotiation.cc:299] Negotiation successful
0926 11:30:20.520658 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 11:30:20.521459 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 7978
I20260926 11:30:20.521590 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:30:20.521775 8002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:20.507940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36439 (local address 127.0.98.1:53283)
0926 11:30:20.508426 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:20.508468 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:20.509458 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:20.510171 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:20.510183 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:20.510606 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:20.511344 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:20.511369 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:20.512483 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:20.512487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:20.513596 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:20.513605 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:20.514239 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:20.514243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:20.514423 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:20.515025 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:20.515041 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:20.516673 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:20.518662 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:20.518669 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:20.518683 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:20.518973 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:20.519294 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:20.519299 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:20.519302 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:20.519477 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:20.519970 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:20.519976 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:20.520264 (+ 288us) client_negotiation.cc:769] Sending connection context
0926 11:30:20.520474 (+ 210us) client_negotiation.cc:240] Negotiation successful
0926 11:30:20.520742 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":138,"threads_started":1}
I20260926 11:30:20.521915 392 external_mini_cluster.cc:1428] Setting key fa7930d5f723f8bfafb4d21d7909eb3d
I20260926 11:30:20.523399 8109 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36439
I20260926 11:30:20.523612 8109 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:20.524324 8109 heartbeater.cc:511] Master 127.0.98.62:36439 requested a full tablet report, sending...
I20260926 11:30:20.526050 7904 ts_manager.cc:194] Registered new tserver with Master: 019b6a6a8c374064be3f9293e1b62077 (127.0.98.1:41563)
I20260926 11:30:20.527163 7904 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53283
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260926 11:30:20.534431 8116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:20.525633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36439 (local address 127.0.98.1:33103)
0926 11:30:20.527850 (+ 2217us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:20.527864 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:20.527980 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:20.528271 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:20.528274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:20.528480 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:20.528575 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:20.528580 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:20.529311 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:20.529314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:20.529857 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:20.529864 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:20.530315 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:20.530319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:20.530411 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:20.530930 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:20.530942 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:20.531251 (+ 309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:20.532898 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:20.532902 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:20.532904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:20.533093 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:20.533397 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:20.533400 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:20.533407 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:20.533479 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:20.533830 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:20.533833 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:20.533925 (+ 92us) client_negotiation.cc:769] Sending connection context
0926 11:30:20.534034 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 11:30:20.534205 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2081,"thread_start_us":149,"threads_started":1}
I20260926 11:30:20.534487 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:20.525695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33103 (local address 127.0.98.62:36439)
0926 11:30:20.525824 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:20.525830 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:20.527955 (+ 2125us) server_negotiation.cc:407] Connection header received
0926 11:30:20.528123 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:20.528126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:20.528168 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:20.528255 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:20.528681 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:20.529181 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:20.529967 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:20.530199 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:20.530303 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:20.531392 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:20.531396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:20.531399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:20.531422 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:20.532781 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:20.533191 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:20.533195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:20.533196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:20.533268 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:20.533565 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:20.533570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:20.533572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:20.533732 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:20.533841 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:20.534171 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 11:30:20.534302 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260926 11:30:20.548859 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36439
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:20.656936 8118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:20.657229 8118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:20.657320 8118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:20.661370 8118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:20.661455 8118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:20.661526 8118 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 11:30:20.661602 8118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:30:20.666834 8118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:20.667941 8118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:20.668919 8118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:20.678277 8124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:20.678303 8123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:20.678339 8126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:20.678481 8118 server_base.cc:1034] running on GCE node
I20260926 11:30:20.678982 8118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:20.679574 8118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:20.680727 8118 hybrid_clock.cc:648] HybridClock initialized: now 1790422220680705 us; error 32 us; skew 500 ppm
Sep 26 11:30:20 dist-test-slave-mrck krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422220, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:20.683549 8118 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 11:30:20.684504 8118 webserver.cc:460] Webserver started at http://127.0.98.2:37679/ using document root <none> and password file <none>
I20260926 11:30:20.685039 8118 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:20.685098 8118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:20.685321 8118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:20.687279 8118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-1/data/instance:
uuid: "604a0651b0d94c288f6dd4f7548f3c29"
format_stamp: "Formatted at 2026-09-26 11:30:20 on dist-test-slave-mrck"
server_key: "1729836618709b296d698f4faa8069ca"
server_key_iv: "44598d3fb6b6fcf0bbb6531a41b8d94f"
server_key_version: "encryptionkey@0"
I20260926 11:30:20.687865 8118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "604a0651b0d94c288f6dd4f7548f3c29"
format_stamp: "Formatted at 2026-09-26 11:30:20 on dist-test-slave-mrck"
server_key: "1729836618709b296d698f4faa8069ca"
server_key_iv: "44598d3fb6b6fcf0bbb6531a41b8d94f"
server_key_version: "encryptionkey@0"
I20260926 11:30:20.691335 8118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:30:20.693603 8133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:20.694815 8118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:30:20.694943 8118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "604a0651b0d94c288f6dd4f7548f3c29"
format_stamp: "Formatted at 2026-09-26 11:30:20 on dist-test-slave-mrck"
server_key: "1729836618709b296d698f4faa8069ca"
server_key_iv: "44598d3fb6b6fcf0bbb6531a41b8d94f"
server_key_version: "encryptionkey@0"
I20260926 11:30:20.695037 8118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:20.711604 8118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:20.716326 8118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:20.716539 8118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:20.718530 8118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:20.718596 8118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:20.718649 8118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:20.718684 8118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:20.729846 8118 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36991
I20260926 11:30:20.731066 8118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:30:20.732846 8248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36991 every 8 connection(s)
Sep 26 11:30:20 dist-test-slave-mrck krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422220, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 11:30:20.737980 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:20.719389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44961 (local address 127.0.98.62:36439)
0926 11:30:20.719545 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:20.719551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:20.720595 (+ 1044us) server_negotiation.cc:407] Connection header received
0926 11:30:20.721951 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:20.721955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:20.722023 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:20.722091 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:20.729807 (+ 7716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:20.730418 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:20.731977 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:20.732234 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:20.732355 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:20.734995 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:20.734998 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:20.735000 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:20.735025 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:20.736295 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:20.736739 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:20.736741 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:20.736743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:20.736811 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:20.737085 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:20.737088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:20.737089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:20.737249 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:20.737331 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:20.737765 (+ 434us) server_negotiation.cc:299] Negotiation successful
0926 11:30:20.737862 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 11:30:20.738785 8142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:20.719805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36439 (local address 127.0.98.2:44961)
0926 11:30:20.720426 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:20.720489 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:20.721679 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:20.722546 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:20.722558 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:20.728285 (+ 5727us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:20.729592 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:20.729621 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:20.730555 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:20.730558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:20.731840 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:20.731853 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:20.732394 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:20.732398 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:20.732607 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:20.733314 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:20.733339 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:20.734828 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:20.736394 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:20.736399 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:20.736410 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:20.736651 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:20.736880 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:20.736882 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:20.736884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:20.737009 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:20.737339 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:20.737344 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:20.737567 (+ 223us) client_negotiation.cc:769] Sending connection context
0926 11:30:20.737714 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 11:30:20.737962 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":159,"threads_started":1}
I20260926 11:30:20.739236 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 8118
I20260926 11:30:20.739382 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:30:20.739708 392 external_mini_cluster.cc:1428] Setting key 3d03a94c325ab1034743a56580aa43e0
I20260926 11:30:20.740221 8249 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36439
I20260926 11:30:20.740371 8249 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:20.740837 8249 heartbeater.cc:511] Master 127.0.98.62:36439 requested a full tablet report, sending...
I20260926 11:30:20.741828 7905 ts_manager.cc:194] Registered new tserver with Master: 604a0651b0d94c288f6dd4f7548f3c29 (127.0.98.2:36991)
I20260926 11:30:20.742383 7905 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44961
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 11:30:20.750379 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:20.743222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43181 (local address 127.0.98.62:36439)
0926 11:30:20.743326 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:20.743330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:20.743566 (+ 236us) server_negotiation.cc:407] Connection header received
0926 11:30:20.743664 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:20.743668 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:20.743775 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:20.743864 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:20.744325 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:20.745184 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:20.746008 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:20.746231 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:20.746341 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:20.747418 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:20.747422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:20.747424 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:20.747451 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:20.748839 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:20.749226 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:20.749232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:20.749234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:20.749312 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:20.749568 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:20.749573 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:20.749575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:20.749765 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:20.749868 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:20.750114 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 11:30:20.750217 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 11:30:20.750386 8256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:20.743163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36439 (local address 127.0.98.2:43181)
0926 11:30:20.743471 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:20.743485 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:20.743581 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:20.743883 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:20.743886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:20.744098 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:20.744206 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:20.744213 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:20.745311 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:20.745315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:20.745896 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:20.745903 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:20.746343 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:20.746346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:20.746442 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:20.746973 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:20.746984 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:20.747294 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:20.748942 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:20.748946 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:20.748948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:20.749136 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:20.749398 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:20.749403 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:20.749405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:20.749485 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:20.749857 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:20.749860 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:20.749951 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 11:30:20.750050 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 11:30:20.750188 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":102,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 11:30:20.767798 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36439
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:20.879776 8258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:20.880085 8258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:20.880133 8258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:20.884449 8258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:20.884524 8258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:20.884570 8258 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 11:30:20.884605 8258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:30:20.890208 8258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:36439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:20.891363 8258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:20.892472 8258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:20.902679 8266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:20.902688 8263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:20.902731 8258 server_base.cc:1034] running on GCE node
W20260926 11:30:20.902693 8264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:20.903398 8258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:20.904115 8258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:20.905285 8258 hybrid_clock.cc:648] HybridClock initialized: now 1790422220905267 us; error 34 us; skew 500 ppm
Sep 26 11:30:20 dist-test-slave-mrck krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422220, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:20.908088 8258 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 11:30:20.908990 8258 webserver.cc:460] Webserver started at http://127.0.98.3:33393/ using document root <none> and password file <none>
I20260926 11:30:20.909590 8258 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:20.909682 8258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:20.909914 8258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:20.912073 8258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-2/data/instance:
uuid: "a4a63e8498964e47b64a3d761292bd49"
format_stamp: "Formatted at 2026-09-26 11:30:20 on dist-test-slave-mrck"
server_key: "36b04b09d8a4e989b94036d7774c085f"
server_key_iv: "258381032c2b8799cd7b926900c9e9b7"
server_key_version: "encryptionkey@0"
I20260926 11:30:20.912653 8258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a4a63e8498964e47b64a3d761292bd49"
format_stamp: "Formatted at 2026-09-26 11:30:20 on dist-test-slave-mrck"
server_key: "36b04b09d8a4e989b94036d7774c085f"
server_key_iv: "258381032c2b8799cd7b926900c9e9b7"
server_key_version: "encryptionkey@0"
I20260926 11:30:20.916476 8258 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 11:30:20.918900 8273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:20.920131 8258 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:30:20.920271 8258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "a4a63e8498964e47b64a3d761292bd49"
format_stamp: "Formatted at 2026-09-26 11:30:20 on dist-test-slave-mrck"
server_key: "36b04b09d8a4e989b94036d7774c085f"
server_key_iv: "258381032c2b8799cd7b926900c9e9b7"
server_key_version: "encryptionkey@0"
I20260926 11:30:20.920378 8258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:20.941329 8258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:20.944298 8258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:20.944494 8258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:20.945966 8258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:20.946034 8258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:20.946100 8258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:20.946131 8258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 11:30:20 dist-test-slave-mrck krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422220, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 11:30:20.956748 8258 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32989
I20260926 11:30:20.956929 8388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32989 every 8 connection(s)
I20260926 11:30:20.957723 8258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:30:20.959908 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:20.946744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33613 (local address 127.0.98.62:36439)
0926 11:30:20.946886 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:20.946890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:20.947798 (+ 908us) server_negotiation.cc:407] Connection header received
0926 11:30:20.948924 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:20.948928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:20.948985 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:20.949085 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:20.950632 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:20.951546 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:20.952997 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:20.953273 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:20.953389 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:20.956172 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:20.956176 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:20.956179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:20.956212 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:20.957897 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:20.958525 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:20.958528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:20.958530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:20.958621 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:20.958944 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:20.958946 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:20.958947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:20.959118 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:20.959195 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:20.959620 (+ 425us) server_negotiation.cc:299] Negotiation successful
0926 11:30:20.959717 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 11:30:20.960673 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:20.947180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36439 (local address 127.0.98.3:33613)
0926 11:30:20.947618 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:20.947670 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:20.948689 (+ 1019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:20.949288 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:20.949295 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:20.949804 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:20.950465 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:20.950483 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:20.951690 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:20.951693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:20.952875 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:20.952884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:20.953441 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:20.953445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:20.953648 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:20.954306 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:20.954322 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:20.956005 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:20.958001 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:20.958011 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:20.958029 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:20.958383 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:20.958723 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:20.958725 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:20.958727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:20.958853 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:20.959198 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:20.959203 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:20.959419 (+ 216us) client_negotiation.cc:769] Sending connection context
0926 11:30:20.959562 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 11:30:20.959876 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":126,"threads_started":1}
I20260926 11:30:20.961760 8389 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36439
I20260926 11:30:20.961910 8389 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:20.962391 8389 heartbeater.cc:511] Master 127.0.98.62:36439 requested a full tablet report, sending...
I20260926 11:30:20.963379 7904 ts_manager.cc:194] Registered new tserver with Master: a4a63e8498964e47b64a3d761292bd49 (127.0.98.3:32989)
I20260926 11:30:20.964129 7904 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33613
I20260926 11:30:20.964143 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 8258
I20260926 11:30:20.964306 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:30:20.964646 392 external_mini_cluster.cc:1428] Setting key 1c9a6123f28ec3a3936a1cfd5d662275
I20260926 11:30:20.966539 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:30:20.971380 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:20.964008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49429 (local address 127.0.98.62:36439)
0926 11:30:20.964105 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:20.964109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:20.964587 (+ 478us) server_negotiation.cc:407] Connection header received
0926 11:30:20.964700 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:20.964706 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:20.964752 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:20.964831 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:20.965538 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:20.966250 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:20.967074 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:20.967300 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:20.967399 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:20.968444 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:20.968448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:20.968450 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:20.968472 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:20.969852 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:20.970256 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:20.970259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:20.970261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:20.970339 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:20.970609 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:20.970612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:20.970614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:20.970783 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:20.970881 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:20.971126 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 11:30:20.971256 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 11:30:20.971390 8395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:20.963965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36439 (local address 127.0.98.3:49429)
0926 11:30:20.964507 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:20.964519 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:20.964625 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:20.964844 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:20.964847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:20.965282 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:20.965441 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:20.965450 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:20.966375 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:20.966378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:20.966960 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:20.966968 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:20.967401 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:20.967405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:20.967495 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:20.968011 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:20.968021 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:20.968313 (+ 292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:20.969955 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:20.969960 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:20.969962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:20.970157 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:20.970433 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:20.970436 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:20.970438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:20.970518 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:20.970870 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:20.970873 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:20.970967 (+ 94us) client_negotiation.cc:769] Sending connection context
0926 11:30:20.971066 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 11:30:20.971209 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":93,"threads_started":1}
Sep 26 11:30:20 dist-test-slave-mrck krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422220, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 11:30:20 dist-test-slave-mrck krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:20 dist-test-slave-mrck krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 11:30:20.990896 7955 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:20.984744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34266 (local address 127.0.98.62:36439)
0926 11:30:20.984867 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:20.984871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:20.984883 (+ 12us) server_negotiation.cc:407] Connection header received
0926 11:30:20.984924 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:20.984926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:20.984963 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:20.985034 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:20.985716 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:20.986274 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:20.987166 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:20.987389 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:20.987482 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:20.990742 (+ 3260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34266: BlockingRecv error: recv got EOF from 127.0.0.1:34266 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 26 11:30:20 dist-test-slave-mrck krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422220, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 11:30:21.000437 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:20.992295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34270 (local address 127.0.98.62:36439)
0926 11:30:20.992431 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:20.992435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:20.992541 (+ 106us) server_negotiation.cc:407] Connection header received
0926 11:30:20.992688 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:20.992695 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:20.992735 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:20.992812 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:20.993377 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:20.993882 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:20.994656 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:20.994872 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:20.994976 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:20.997156 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:20.997164 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:20.997166 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:20.997191 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:20.998472 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:20.998936 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:20.998940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:20.998942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:20.999028 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:20.999337 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:20.999343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:20.999346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:20.999581 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:20.999720 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:21.000055 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 11:30:21.000212 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 11:30:21.003551 7904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:30:21.006561 7904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:30:21.020776 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:21.014794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60964 (local address 127.0.98.1:41563)
0926 11:30:21.016235 (+ 1441us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:21.016251 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:21.016275 (+ 24us) server_negotiation.cc:407] Connection header received
0926 11:30:21.016349 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:21.016353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:21.016513 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:21.016668 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:30:21.016943 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.017661 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.019324 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.020081 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.020198 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:21.020390 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:21.020506 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 11:30:21.020606 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1343,"thread_start_us":98,"threads_started":1}
I20260926 11:30:21.020815 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:21.014722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36991 (local address 127.0.0.1:35382)
0926 11:30:21.015465 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:21.015516 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:21.015651 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:21.016406 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:21.016412 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:21.016437 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:30:21.016579 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:21.016586 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.018060 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.018063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:21.019210 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:21.019218 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.020373 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.020377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:21.020464 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:21.020485 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 11:30:21.020542 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 11:30:21.020621 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"mutex_wait_us":45,"spinlock_wait_cycles":10240,"thread_start_us":129,"threads_started":1}
I20260926 11:30:21.020777 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:21.014651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41563 (local address 127.0.0.1:60964)
0926 11:30:21.015463 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:21.015513 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:21.015651 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:21.016659 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:21.016663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:21.016688 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:30:21.016840 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:21.016845 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.017782 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.017785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:21.019183 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:21.019193 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.020196 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.020199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:21.020307 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:21.020345 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 11:30:21.020475 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 11:30:21.020607 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":27,"spinlock_wait_cycles":5760,"thread_start_us":75,"threads_started":1}
I20260926 11:30:21.021112 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:21.015255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46356 (local address 127.0.98.3:32989)
0926 11:30:21.015961 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:21.015973 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:21.015995 (+ 22us) server_negotiation.cc:407] Connection header received
0926 11:30:21.016135 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:21.016143 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:21.016379 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:21.016525 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:30:21.016980 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.018095 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.019402 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.020232 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.020356 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:21.020519 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:21.020863 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 11:30:21.020965 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":104,"threads_started":1}
I20260926 11:30:21.021371 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:21.014988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32989 (local address 127.0.0.1:46356)
0926 11:30:21.015922 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:21.015936 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:21.016008 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:21.016656 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:21.016658 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:21.016693 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:30:21.016852 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:21.016860 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.018196 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.018201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:21.019291 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:21.019303 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.020574 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.020577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:21.020648 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:21.020668 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 11:30:21.020733 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 11:30:21.020792 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"mutex_wait_us":42,"spinlock_wait_cycles":8704,"thread_start_us":86,"threads_started":1}
I20260926 11:30:21.021540 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:21.015423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35382 (local address 127.0.98.2:36991)
0926 11:30:21.015865 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:21.015878 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:21.015899 (+ 21us) server_negotiation.cc:407] Connection header received
0926 11:30:21.015979 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:21.015984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:21.016196 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:21.016400 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:30:21.016711 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.017948 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.019348 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.020232 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.021065 (+ 833us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:21.021238 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:21.021290 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 11:30:21.021380 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":108,"threads_started":1}
I20260926 11:30:21.024235 8043 tablet_service.cc:1467] Processing CreateTablet for tablet 0939e49885e1421782c440fd7319f106 (DEFAULT_TABLE table=test-table [id=ba9aa37448f14b4c82c7cad2af19ed14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:30:21.024245 8165 tablet_service.cc:1467] Processing CreateTablet for tablet 0939e49885e1421782c440fd7319f106 (DEFAULT_TABLE table=test-table [id=ba9aa37448f14b4c82c7cad2af19ed14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:30:21.024259 8323 tablet_service.cc:1467] Processing CreateTablet for tablet 0939e49885e1421782c440fd7319f106 (DEFAULT_TABLE table=test-table [id=ba9aa37448f14b4c82c7cad2af19ed14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:30:21.025409 8165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0939e49885e1421782c440fd7319f106. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:21.025449 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0939e49885e1421782c440fd7319f106. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:21.025424 8323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0939e49885e1421782c440fd7319f106. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:21.032814 8417 tablet_bootstrap.cc:492] T 0939e49885e1421782c440fd7319f106 P a4a63e8498964e47b64a3d761292bd49: Bootstrap starting.
I20260926 11:30:21.033876 8418 tablet_bootstrap.cc:492] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077: Bootstrap starting.
I20260926 11:30:21.034190 8419 tablet_bootstrap.cc:492] T 0939e49885e1421782c440fd7319f106 P 604a0651b0d94c288f6dd4f7548f3c29: Bootstrap starting.
I20260926 11:30:21.035210 8417 tablet_bootstrap.cc:654] T 0939e49885e1421782c440fd7319f106 P a4a63e8498964e47b64a3d761292bd49: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:21.036039 8417 log.cc:824] T 0939e49885e1421782c440fd7319f106 P a4a63e8498964e47b64a3d761292bd49: Log is configured to *not* fsync() on all Append() calls
I20260926 11:30:21.036154 8418 tablet_bootstrap.cc:654] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:21.036154 8419 tablet_bootstrap.cc:654] T 0939e49885e1421782c440fd7319f106 P 604a0651b0d94c288f6dd4f7548f3c29: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:21.037039 8419 log.cc:824] T 0939e49885e1421782c440fd7319f106 P 604a0651b0d94c288f6dd4f7548f3c29: Log is configured to *not* fsync() on all Append() calls
I20260926 11:30:21.037662 8418 log.cc:824] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077: Log is configured to *not* fsync() on all Append() calls
I20260926 11:30:21.038120 8417 tablet_bootstrap.cc:492] T 0939e49885e1421782c440fd7319f106 P a4a63e8498964e47b64a3d761292bd49: No bootstrap required, opened a new log
I20260926 11:30:21.038329 8417 ts_tablet_manager.cc:1397] T 0939e49885e1421782c440fd7319f106 P a4a63e8498964e47b64a3d761292bd49: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 11:30:21.039017 8419 tablet_bootstrap.cc:492] T 0939e49885e1421782c440fd7319f106 P 604a0651b0d94c288f6dd4f7548f3c29: No bootstrap required, opened a new log
I20260926 11:30:21.039206 8419 ts_tablet_manager.cc:1397] T 0939e49885e1421782c440fd7319f106 P 604a0651b0d94c288f6dd4f7548f3c29: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 11:30:21.039438 8418 tablet_bootstrap.cc:492] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077: No bootstrap required, opened a new log
I20260926 11:30:21.039614 8418 ts_tablet_manager.cc:1397] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 11:30:21.041700 8417 raft_consensus.cc:348] T 0939e49885e1421782c440fd7319f106 P a4a63e8498964e47b64a3d761292bd49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.041954 8417 raft_consensus.cc:374] T 0939e49885e1421782c440fd7319f106 P a4a63e8498964e47b64a3d761292bd49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:21.042014 8417 raft_consensus.cc:729] T 0939e49885e1421782c440fd7319f106 P a4a63e8498964e47b64a3d761292bd49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4a63e8498964e47b64a3d761292bd49, State: Initialized, Role: FOLLOWER
I20260926 11:30:21.042469 8417 consensus_queue.cc:260] T 0939e49885e1421782c440fd7319f106 P a4a63e8498964e47b64a3d761292bd49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.043149 8389 heartbeater.cc:503] Master 127.0.98.62:36439 was elected leader, sending a full tablet report...
I20260926 11:30:21.043205 8419 raft_consensus.cc:348] T 0939e49885e1421782c440fd7319f106 P 604a0651b0d94c288f6dd4f7548f3c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.043470 8419 raft_consensus.cc:374] T 0939e49885e1421782c440fd7319f106 P 604a0651b0d94c288f6dd4f7548f3c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:21.043502 8417 ts_tablet_manager.cc:1428] T 0939e49885e1421782c440fd7319f106 P a4a63e8498964e47b64a3d761292bd49: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 11:30:21.043531 8419 raft_consensus.cc:729] T 0939e49885e1421782c440fd7319f106 P 604a0651b0d94c288f6dd4f7548f3c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 604a0651b0d94c288f6dd4f7548f3c29, State: Initialized, Role: FOLLOWER
I20260926 11:30:21.043980 8419 consensus_queue.cc:260] T 0939e49885e1421782c440fd7319f106 P 604a0651b0d94c288f6dd4f7548f3c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.044122 8418 raft_consensus.cc:348] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.044379 8418 raft_consensus.cc:374] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:21.044435 8418 raft_consensus.cc:729] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 019b6a6a8c374064be3f9293e1b62077, State: Initialized, Role: FOLLOWER
I20260926 11:30:21.044862 8418 consensus_queue.cc:260] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.045492 8109 heartbeater.cc:503] Master 127.0.98.62:36439 was elected leader, sending a full tablet report...
I20260926 11:30:21.045603 8419 ts_tablet_manager.cc:1428] T 0939e49885e1421782c440fd7319f106 P 604a0651b0d94c288f6dd4f7548f3c29: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 11:30:21.045938 8249 heartbeater.cc:503] Master 127.0.98.62:36439 was elected leader, sending a full tablet report...
I20260926 11:30:21.046749 8418 ts_tablet_manager.cc:1428] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 11:30:21.137450 8425 raft_consensus.cc:482] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:30:21.137669 8425 raft_consensus.cc:504] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.138792 8425 leader_election.cc:288] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 604a0651b0d94c288f6dd4f7548f3c29 (127.0.98.2:36991), a4a63e8498964e47b64a3d761292bd49 (127.0.98.3:32989)
I20260926 11:30:21.142957 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:21.139271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41979 (local address 127.0.98.3:32989)
0926 11:30:21.139412 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:21.139416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:21.139547 (+ 131us) server_negotiation.cc:407] Connection header received
0926 11:30:21.139696 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:21.139703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:21.139786 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:21.139865 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:30:21.140199 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.140744 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.142023 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.142423 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.142532 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:21.142593 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:21.142759 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 11:30:21.142823 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 11:30:21.143023 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:21.139062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32989 (local address 127.0.98.1:41979)
0926 11:30:21.139459 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:21.139472 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:21.139600 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:21.139881 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:21.139884 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:21.139899 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:30:21.140100 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:21.140106 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.140847 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.140858 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:21.141911 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:21.141920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.142537 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.142541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:21.142636 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:21.142658 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 11:30:21.142746 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 11:30:21.142817 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":101,"thread_start_us":112,"threads_started":1}
I20260926 11:30:21.143013 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:21.139174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48665 (local address 127.0.98.2:36991)
0926 11:30:21.139321 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:21.139326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:21.139635 (+ 309us) server_negotiation.cc:407] Connection header received
0926 11:30:21.139784 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:21.139787 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:21.139835 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:21.139914 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:30:21.140251 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.140747 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.142023 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.142453 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.142579 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:21.142641 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:21.142815 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 11:30:21.142882 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 11:30:21.143460 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:21.139062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36991 (local address 127.0.98.1:48665)
0926 11:30:21.139536 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:21.139547 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:21.139669 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:21.139924 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:21.139926 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:21.139941 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:30:21.140120 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:21.140126 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.140862 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.140866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:21.141907 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:21.141916 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.142582 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.142585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:21.142697 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:21.142720 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 11:30:21.142801 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 11:30:21.142869 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":126,"thread_start_us":152,"threads_started":1}
I20260926 11:30:21.143587 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0939e49885e1421782c440fd7319f106" candidate_uuid: "019b6a6a8c374064be3f9293e1b62077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4a63e8498964e47b64a3d761292bd49" is_pre_election: true
I20260926 11:30:21.143970 8343 raft_consensus.cc:2454] T 0939e49885e1421782c440fd7319f106 P a4a63e8498964e47b64a3d761292bd49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 019b6a6a8c374064be3f9293e1b62077 in term 0.
I20260926 11:30:21.144099 8203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0939e49885e1421782c440fd7319f106" candidate_uuid: "019b6a6a8c374064be3f9293e1b62077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "604a0651b0d94c288f6dd4f7548f3c29" is_pre_election: true
I20260926 11:30:21.144385 8203 raft_consensus.cc:2454] T 0939e49885e1421782c440fd7319f106 P 604a0651b0d94c288f6dd4f7548f3c29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 019b6a6a8c374064be3f9293e1b62077 in term 0.
I20260926 11:30:21.144503 7997 leader_election.cc:302] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 019b6a6a8c374064be3f9293e1b62077, a4a63e8498964e47b64a3d761292bd49; no voters:
I20260926 11:30:21.144799 8425 raft_consensus.cc:2781] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:30:21.144891 8425 raft_consensus.cc:482] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:30:21.144943 8425 raft_consensus.cc:3037] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:21.145962 8425 raft_consensus.cc:504] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.146296 8425 leader_election.cc:288] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077 [CANDIDATE]: Term 1 election: Requested vote from peers 604a0651b0d94c288f6dd4f7548f3c29 (127.0.98.2:36991), a4a63e8498964e47b64a3d761292bd49 (127.0.98.3:32989)
I20260926 11:30:21.146596 8203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0939e49885e1421782c440fd7319f106" candidate_uuid: "019b6a6a8c374064be3f9293e1b62077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "604a0651b0d94c288f6dd4f7548f3c29"
I20260926 11:30:21.146653 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0939e49885e1421782c440fd7319f106" candidate_uuid: "019b6a6a8c374064be3f9293e1b62077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4a63e8498964e47b64a3d761292bd49"
I20260926 11:30:21.146737 8203 raft_consensus.cc:3037] T 0939e49885e1421782c440fd7319f106 P 604a0651b0d94c288f6dd4f7548f3c29 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:21.146798 8343 raft_consensus.cc:3037] T 0939e49885e1421782c440fd7319f106 P a4a63e8498964e47b64a3d761292bd49 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:21.147818 8203 raft_consensus.cc:2454] T 0939e49885e1421782c440fd7319f106 P 604a0651b0d94c288f6dd4f7548f3c29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 019b6a6a8c374064be3f9293e1b62077 in term 1.
I20260926 11:30:21.147923 8343 raft_consensus.cc:2454] T 0939e49885e1421782c440fd7319f106 P a4a63e8498964e47b64a3d761292bd49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 019b6a6a8c374064be3f9293e1b62077 in term 1.
I20260926 11:30:21.148131 7994 leader_election.cc:302] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 019b6a6a8c374064be3f9293e1b62077, 604a0651b0d94c288f6dd4f7548f3c29; no voters:
I20260926 11:30:21.148288 8425 raft_consensus.cc:2781] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:30:21.148520 8425 raft_consensus.cc:686] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077 [term 1 LEADER]: Becoming Leader. State: Replica: 019b6a6a8c374064be3f9293e1b62077, State: Running, Role: LEADER
I20260926 11:30:21.148831 8425 consensus_queue.cc:237] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.151871 7905 catalog_manager.cc:5495] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077 reported cstate change: term changed from 0 to 1, leader changed from <none> to 019b6a6a8c374064be3f9293e1b62077 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "019b6a6a8c374064be3f9293e1b62077" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } health_report { overall_health: UNKNOWN } } }
I20260926 11:30:21.174233 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:21.170712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60976 (local address 127.0.98.1:41563)
0926 11:30:21.170860 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:21.170866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:21.170908 (+ 42us) server_negotiation.cc:407] Connection header received
0926 11:30:21.170988 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:21.170990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:21.171045 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:21.171152 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:30:21.171422 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.171967 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.172808 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.173031 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.173141 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:21.173410 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:30:21.173848 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:30:21.173981 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:21.174028 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 11:30:21.174086 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 11:30:21.180337 8203 raft_consensus.cc:1261] T 0939e49885e1421782c440fd7319f106 P 604a0651b0d94c288f6dd4f7548f3c29 [term 1 FOLLOWER]: Refusing update from remote peer 019b6a6a8c374064be3f9293e1b62077: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:30:21.180847 8343 raft_consensus.cc:1261] T 0939e49885e1421782c440fd7319f106 P a4a63e8498964e47b64a3d761292bd49 [term 1 FOLLOWER]: Refusing update from remote peer 019b6a6a8c374064be3f9293e1b62077: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:30:21.181051 8428 consensus_queue.cc:1035] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:30:21.181450 8425 consensus_queue.cc:1035] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:30:21.192330 8433 mvcc.cc:205] Tried to move back new op lower bound from 7333569417947090944 to 7333569417828274176. Current Snapshot: MvccSnapshot[applied={T|T < 7333569417947090944}]
I20260926 11:30:21.197247 7905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:30:21.197396 7905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:30:21.199759 7905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c53348dae04456b9c59629c03d412d6: Sending DeleteTablet for 3 replicas of tablet 0939e49885e1421782c440fd7319f106
I20260926 11:30:21.200632 8165 tablet_service.cc:1514] Processing DeleteTablet for tablet 0939e49885e1421782c440fd7319f106 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:30:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35382
I20260926 11:30:21.200646 8323 tablet_service.cc:1514] Processing DeleteTablet for tablet 0939e49885e1421782c440fd7319f106 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:30:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46356
I20260926 11:30:21.201130 8442 tablet_replica.cc:330] T 0939e49885e1421782c440fd7319f106 P a4a63e8498964e47b64a3d761292bd49: stopping tablet replica
I20260926 11:30:21.201501 8442 raft_consensus.cc:2229] T 0939e49885e1421782c440fd7319f106 P a4a63e8498964e47b64a3d761292bd49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:30:21.201556 8043 tablet_service.cc:1514] Processing DeleteTablet for tablet 0939e49885e1421782c440fd7319f106 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:30:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60964
I20260926 11:30:21.201807 8442 raft_consensus.cc:2258] T 0939e49885e1421782c440fd7319f106 P a4a63e8498964e47b64a3d761292bd49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:30:21.202687 7905 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34270:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:30:21.203088 7905 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:30:21.203313 8442 ts_tablet_manager.cc:1905] T 0939e49885e1421782c440fd7319f106 P a4a63e8498964e47b64a3d761292bd49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:30:21.204870 8441 tablet_replica.cc:330] T 0939e49885e1421782c440fd7319f106 P 604a0651b0d94c288f6dd4f7548f3c29: stopping tablet replica
I20260926 11:30:21.205240 8441 raft_consensus.cc:2229] T 0939e49885e1421782c440fd7319f106 P 604a0651b0d94c288f6dd4f7548f3c29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:30:21.206079 8442 ts_tablet_manager.cc:1918] T 0939e49885e1421782c440fd7319f106 P a4a63e8498964e47b64a3d761292bd49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:30:21.206178 8442 log.cc:1196] T 0939e49885e1421782c440fd7319f106 P a4a63e8498964e47b64a3d761292bd49: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-2/wal/wals/0939e49885e1421782c440fd7319f106
I20260926 11:30:21.206496 8442 ts_tablet_manager.cc:1939] T 0939e49885e1421782c440fd7319f106 P a4a63e8498964e47b64a3d761292bd49: Deleting consensus metadata
I20260926 11:30:21.207691 7893 catalog_manager.cc:4841] TS a4a63e8498964e47b64a3d761292bd49 (127.0.98.3:32989): tablet 0939e49885e1421782c440fd7319f106 (table test-table [id=ba9aa37448f14b4c82c7cad2af19ed14]) successfully deleted
I20260926 11:30:21.208009 8443 tablet_replica.cc:330] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077: stopping tablet replica
I20260926 11:30:21.208297 8443 raft_consensus.cc:2229] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:30:21.208595 8443 raft_consensus.cc:2258] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:30:21.208809 8043 tablet_service.cc:1467] Processing CreateTablet for tablet 20d9e0ed6bfe4f508646fb170a17996c (DEFAULT_TABLE table=test-table [id=1e22bbe924a64e9c9722cd0326b04f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:30:21.209077 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20d9e0ed6bfe4f508646fb170a17996c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:21.210002 8443 ts_tablet_manager.cc:1905] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:30:21.211432 8418 tablet_bootstrap.cc:492] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077: Bootstrap starting.
I20260926 11:30:21.212105 8323 tablet_service.cc:1467] Processing CreateTablet for tablet 20d9e0ed6bfe4f508646fb170a17996c (DEFAULT_TABLE table=test-table [id=1e22bbe924a64e9c9722cd0326b04f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:30:21.212312 8443 ts_tablet_manager.cc:1918] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:30:21.212361 8323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20d9e0ed6bfe4f508646fb170a17996c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:21.212389 8443 log.cc:1196] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-0/wal/wals/0939e49885e1421782c440fd7319f106
I20260926 11:30:21.212411 8418 tablet_bootstrap.cc:654] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:21.212662 8443 ts_tablet_manager.cc:1939] T 0939e49885e1421782c440fd7319f106 P 019b6a6a8c374064be3f9293e1b62077: Deleting consensus metadata
I20260926 11:30:21.213434 8441 raft_consensus.cc:2258] T 0939e49885e1421782c440fd7319f106 P 604a0651b0d94c288f6dd4f7548f3c29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:30:21.214811 8417 tablet_bootstrap.cc:492] T 20d9e0ed6bfe4f508646fb170a17996c P a4a63e8498964e47b64a3d761292bd49: Bootstrap starting.
I20260926 11:30:21.214984 8441 ts_tablet_manager.cc:1905] T 0939e49885e1421782c440fd7319f106 P 604a0651b0d94c288f6dd4f7548f3c29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:30:21.215698 8417 tablet_bootstrap.cc:654] T 20d9e0ed6bfe4f508646fb170a17996c P a4a63e8498964e47b64a3d761292bd49: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:21.215993 8165 tablet_service.cc:1467] Processing CreateTablet for tablet 20d9e0ed6bfe4f508646fb170a17996c (DEFAULT_TABLE table=test-table [id=1e22bbe924a64e9c9722cd0326b04f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:30:21.216225 8165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20d9e0ed6bfe4f508646fb170a17996c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:21.217645 8441 ts_tablet_manager.cc:1918] T 0939e49885e1421782c440fd7319f106 P 604a0651b0d94c288f6dd4f7548f3c29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:30:21.218497 8419 tablet_bootstrap.cc:492] T 20d9e0ed6bfe4f508646fb170a17996c P 604a0651b0d94c288f6dd4f7548f3c29: Bootstrap starting.
I20260926 11:30:21.218747 8441 log.cc:1196] T 0939e49885e1421782c440fd7319f106 P 604a0651b0d94c288f6dd4f7548f3c29: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-1/wal/wals/0939e49885e1421782c440fd7319f106
I20260926 11:30:21.219101 8441 ts_tablet_manager.cc:1939] T 0939e49885e1421782c440fd7319f106 P 604a0651b0d94c288f6dd4f7548f3c29: Deleting consensus metadata
I20260926 11:30:21.219708 8419 tablet_bootstrap.cc:654] T 20d9e0ed6bfe4f508646fb170a17996c P 604a0651b0d94c288f6dd4f7548f3c29: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:21.220307 7891 catalog_manager.cc:4841] TS 019b6a6a8c374064be3f9293e1b62077 (127.0.98.1:41563): tablet 0939e49885e1421782c440fd7319f106 (table test-table [id=ba9aa37448f14b4c82c7cad2af19ed14]) successfully deleted
I20260926 11:30:21.220377 7890 catalog_manager.cc:4841] TS 604a0651b0d94c288f6dd4f7548f3c29 (127.0.98.2:36991): tablet 0939e49885e1421782c440fd7319f106 (table test-table [id=ba9aa37448f14b4c82c7cad2af19ed14]) successfully deleted
I20260926 11:30:21.221091 8418 tablet_bootstrap.cc:492] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077: No bootstrap required, opened a new log
I20260926 11:30:21.221185 8418 ts_tablet_manager.cc:1397] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260926 11:30:21.221271 8419 tablet_bootstrap.cc:492] T 20d9e0ed6bfe4f508646fb170a17996c P 604a0651b0d94c288f6dd4f7548f3c29: No bootstrap required, opened a new log
I20260926 11:30:21.221344 8419 ts_tablet_manager.cc:1397] T 20d9e0ed6bfe4f508646fb170a17996c P 604a0651b0d94c288f6dd4f7548f3c29: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 11:30:21.221732 8417 tablet_bootstrap.cc:492] T 20d9e0ed6bfe4f508646fb170a17996c P a4a63e8498964e47b64a3d761292bd49: No bootstrap required, opened a new log
I20260926 11:30:21.221659 8418 raft_consensus.cc:348] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.221722 8419 raft_consensus.cc:348] T 20d9e0ed6bfe4f508646fb170a17996c P 604a0651b0d94c288f6dd4f7548f3c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.221789 8418 raft_consensus.cc:374] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:21.221808 8419 raft_consensus.cc:374] T 20d9e0ed6bfe4f508646fb170a17996c P 604a0651b0d94c288f6dd4f7548f3c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:21.221809 8417 ts_tablet_manager.cc:1397] T 20d9e0ed6bfe4f508646fb170a17996c P a4a63e8498964e47b64a3d761292bd49: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20260926 11:30:21.221829 8418 raft_consensus.cc:729] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 019b6a6a8c374064be3f9293e1b62077, State: Initialized, Role: FOLLOWER
I20260926 11:30:21.221843 8419 raft_consensus.cc:729] T 20d9e0ed6bfe4f508646fb170a17996c P 604a0651b0d94c288f6dd4f7548f3c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 604a0651b0d94c288f6dd4f7548f3c29, State: Initialized, Role: FOLLOWER
I20260926 11:30:21.221959 8418 consensus_queue.cc:260] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.221959 8419 consensus_queue.cc:260] T 20d9e0ed6bfe4f508646fb170a17996c P 604a0651b0d94c288f6dd4f7548f3c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.222215 8418 ts_tablet_manager.cc:1428] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 11:30:21.222223 8419 ts_tablet_manager.cc:1428] T 20d9e0ed6bfe4f508646fb170a17996c P 604a0651b0d94c288f6dd4f7548f3c29: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 11:30:21.222258 8417 raft_consensus.cc:348] T 20d9e0ed6bfe4f508646fb170a17996c P a4a63e8498964e47b64a3d761292bd49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.222355 8417 raft_consensus.cc:374] T 20d9e0ed6bfe4f508646fb170a17996c P a4a63e8498964e47b64a3d761292bd49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:21.222384 8417 raft_consensus.cc:729] T 20d9e0ed6bfe4f508646fb170a17996c P a4a63e8498964e47b64a3d761292bd49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4a63e8498964e47b64a3d761292bd49, State: Initialized, Role: FOLLOWER
I20260926 11:30:21.222500 8417 consensus_queue.cc:260] T 20d9e0ed6bfe4f508646fb170a17996c P a4a63e8498964e47b64a3d761292bd49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.222774 8417 ts_tablet_manager.cc:1428] T 20d9e0ed6bfe4f508646fb170a17996c P a4a63e8498964e47b64a3d761292bd49: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20260926 11:30:21.232302 8250 tablet.cc:2369] T 20d9e0ed6bfe4f508646fb170a17996c P 604a0651b0d94c288f6dd4f7548f3c29: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 11:30:21.270215 8110 tablet.cc:2369] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 11:30:21.458967 8390 tablet.cc:2369] T 20d9e0ed6bfe4f508646fb170a17996c P a4a63e8498964e47b64a3d761292bd49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:30:21.482916 8437 raft_consensus.cc:482] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:30:21.483064 8437 raft_consensus.cc:504] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.483547 8437 leader_election.cc:288] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 604a0651b0d94c288f6dd4f7548f3c29 (127.0.98.2:36991), a4a63e8498964e47b64a3d761292bd49 (127.0.98.3:32989)
I20260926 11:30:21.484026 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20d9e0ed6bfe4f508646fb170a17996c" candidate_uuid: "019b6a6a8c374064be3f9293e1b62077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4a63e8498964e47b64a3d761292bd49" is_pre_election: true
I20260926 11:30:21.484026 8203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20d9e0ed6bfe4f508646fb170a17996c" candidate_uuid: "019b6a6a8c374064be3f9293e1b62077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "604a0651b0d94c288f6dd4f7548f3c29" is_pre_election: true
I20260926 11:30:21.484181 8203 raft_consensus.cc:2454] T 20d9e0ed6bfe4f508646fb170a17996c P 604a0651b0d94c288f6dd4f7548f3c29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 019b6a6a8c374064be3f9293e1b62077 in term 0.
I20260926 11:30:21.484186 8343 raft_consensus.cc:2454] T 20d9e0ed6bfe4f508646fb170a17996c P a4a63e8498964e47b64a3d761292bd49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 019b6a6a8c374064be3f9293e1b62077 in term 0.
I20260926 11:30:21.484472 7997 leader_election.cc:302] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 019b6a6a8c374064be3f9293e1b62077, a4a63e8498964e47b64a3d761292bd49; no voters:
I20260926 11:30:21.484630 8437 raft_consensus.cc:2781] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:30:21.484688 8437 raft_consensus.cc:482] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:30:21.484740 8437 raft_consensus.cc:3037] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:21.485798 8437 raft_consensus.cc:504] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.486146 8437 leader_election.cc:288] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077 [CANDIDATE]: Term 1 election: Requested vote from peers 604a0651b0d94c288f6dd4f7548f3c29 (127.0.98.2:36991), a4a63e8498964e47b64a3d761292bd49 (127.0.98.3:32989)
I20260926 11:30:21.486490 8203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20d9e0ed6bfe4f508646fb170a17996c" candidate_uuid: "019b6a6a8c374064be3f9293e1b62077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "604a0651b0d94c288f6dd4f7548f3c29"
I20260926 11:30:21.486490 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20d9e0ed6bfe4f508646fb170a17996c" candidate_uuid: "019b6a6a8c374064be3f9293e1b62077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4a63e8498964e47b64a3d761292bd49"
I20260926 11:30:21.486610 8343 raft_consensus.cc:3037] T 20d9e0ed6bfe4f508646fb170a17996c P a4a63e8498964e47b64a3d761292bd49 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:21.486610 8203 raft_consensus.cc:3037] T 20d9e0ed6bfe4f508646fb170a17996c P 604a0651b0d94c288f6dd4f7548f3c29 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:21.487684 8343 raft_consensus.cc:2454] T 20d9e0ed6bfe4f508646fb170a17996c P a4a63e8498964e47b64a3d761292bd49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 019b6a6a8c374064be3f9293e1b62077 in term 1.
I20260926 11:30:21.487689 8203 raft_consensus.cc:2454] T 20d9e0ed6bfe4f508646fb170a17996c P 604a0651b0d94c288f6dd4f7548f3c29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 019b6a6a8c374064be3f9293e1b62077 in term 1.
I20260926 11:30:21.487982 7997 leader_election.cc:302] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 019b6a6a8c374064be3f9293e1b62077, a4a63e8498964e47b64a3d761292bd49; no voters:
I20260926 11:30:21.488199 8437 raft_consensus.cc:2781] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:30:21.488271 8437 raft_consensus.cc:686] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077 [term 1 LEADER]: Becoming Leader. State: Replica: 019b6a6a8c374064be3f9293e1b62077, State: Running, Role: LEADER
I20260926 11:30:21.488399 8437 consensus_queue.cc:237] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.489962 7903 catalog_manager.cc:5495] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077 reported cstate change: term changed from 0 to 1, leader changed from <none> to 019b6a6a8c374064be3f9293e1b62077 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "019b6a6a8c374064be3f9293e1b62077" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } health_report { overall_health: UNKNOWN } } }
Sep 26 11:30:21 dist-test-slave-mrck krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422220, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 11:30:21.545485 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:21.537248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36439 (local address 127.0.0.1:34284)
0926 11:30:21.537494 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:21.537512 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:21.537628 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:21.537892 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:21.537894 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:21.538304 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:21.538402 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:21.538409 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.539135 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.539137 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:21.539700 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:21.539708 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.540266 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.540269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:21.540361 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:21.540874 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:21.540886 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:21.542373 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:21.544178 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:21.544185 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:21.544190 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:21.544410 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:21.544640 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:21.544642 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:21.544643 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:21.544714 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:21.545027 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:21.545032 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:21.545123 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 11:30:21.545209 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 11:30:21.545325 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":15,"thread_start_us":65,"threads_started":1}
I20260926 11:30:21.545507 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:21.537365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34284 (local address 127.0.98.62:36439)
0926 11:30:21.537559 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:21.537562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:21.537581 (+ 19us) server_negotiation.cc:407] Connection header received
0926 11:30:21.537731 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:21.537733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:21.537771 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:21.537849 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:21.538506 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.539015 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.539867 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.540151 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.540260 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:21.542506 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:21.542510 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:21.542512 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:21.542538 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:21.544075 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:21.544488 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:21.544491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:21.544492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:21.544562 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:21.544788 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:21.544791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:21.544792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:21.544947 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:21.545022 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:21.545267 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 11:30:21.545376 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":138,"thread_start_us":71,"threads_started":1}
I20260926 11:30:21.549063 7903 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34284:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 11:30:21.549813 7903 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:30:21.554251 8165 tablet_service.cc:1467] Processing CreateTablet for tablet d8ae0659deb541909c73272a5fe13627 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c11b8dfe18a340fe983f7d8ec5746d9a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 11:30:21.554251 8043 tablet_service.cc:1467] Processing CreateTablet for tablet d8ae0659deb541909c73272a5fe13627 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c11b8dfe18a340fe983f7d8ec5746d9a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 11:30:21.554625 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8ae0659deb541909c73272a5fe13627. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:21.554625 8165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8ae0659deb541909c73272a5fe13627. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:21.554574 8323 tablet_service.cc:1467] Processing CreateTablet for tablet d8ae0659deb541909c73272a5fe13627 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c11b8dfe18a340fe983f7d8ec5746d9a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 11:30:21.554869 8323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8ae0659deb541909c73272a5fe13627. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:21.557196 8417 tablet_bootstrap.cc:492] T d8ae0659deb541909c73272a5fe13627 P a4a63e8498964e47b64a3d761292bd49: Bootstrap starting.
I20260926 11:30:21.557344 8418 tablet_bootstrap.cc:492] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077: Bootstrap starting.
I20260926 11:30:21.558166 8418 tablet_bootstrap.cc:654] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:21.558175 8417 tablet_bootstrap.cc:654] T d8ae0659deb541909c73272a5fe13627 P a4a63e8498964e47b64a3d761292bd49: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:21.558784 8419 tablet_bootstrap.cc:492] T d8ae0659deb541909c73272a5fe13627 P 604a0651b0d94c288f6dd4f7548f3c29: Bootstrap starting.
I20260926 11:30:21.559186 8418 tablet_bootstrap.cc:492] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077: No bootstrap required, opened a new log
I20260926 11:30:21.559295 8418 ts_tablet_manager.cc:1397] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 11:30:21.559523 8417 tablet_bootstrap.cc:492] T d8ae0659deb541909c73272a5fe13627 P a4a63e8498964e47b64a3d761292bd49: No bootstrap required, opened a new log
I20260926 11:30:21.559615 8417 ts_tablet_manager.cc:1397] T d8ae0659deb541909c73272a5fe13627 P a4a63e8498964e47b64a3d761292bd49: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 11:30:21.559695 8419 tablet_bootstrap.cc:654] T d8ae0659deb541909c73272a5fe13627 P 604a0651b0d94c288f6dd4f7548f3c29: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:21.559785 8418 raft_consensus.cc:348] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.559914 8418 raft_consensus.cc:374] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:21.559955 8418 raft_consensus.cc:729] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 019b6a6a8c374064be3f9293e1b62077, State: Initialized, Role: FOLLOWER
I20260926 11:30:21.560089 8418 consensus_queue.cc:260] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.560079 8417 raft_consensus.cc:348] T d8ae0659deb541909c73272a5fe13627 P a4a63e8498964e47b64a3d761292bd49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.560201 8417 raft_consensus.cc:374] T d8ae0659deb541909c73272a5fe13627 P a4a63e8498964e47b64a3d761292bd49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:21.560243 8417 raft_consensus.cc:729] T d8ae0659deb541909c73272a5fe13627 P a4a63e8498964e47b64a3d761292bd49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4a63e8498964e47b64a3d761292bd49, State: Initialized, Role: FOLLOWER
I20260926 11:30:21.560392 8418 ts_tablet_manager.cc:1428] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 11:30:21.560369 8417 consensus_queue.cc:260] T d8ae0659deb541909c73272a5fe13627 P a4a63e8498964e47b64a3d761292bd49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.560595 8437 tablet_replica.cc:439] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } } }
I20260926 11:30:21.560699 8437 tablet_replica.cc:442] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:30:21.561126 8423 tablet_replica.cc:439] T d8ae0659deb541909c73272a5fe13627 P a4a63e8498964e47b64a3d761292bd49: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } } }
I20260926 11:30:21.561267 8423 tablet_replica.cc:442] T d8ae0659deb541909c73272a5fe13627 P a4a63e8498964e47b64a3d761292bd49: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:30:21.561529 8419 tablet_bootstrap.cc:492] T d8ae0659deb541909c73272a5fe13627 P 604a0651b0d94c288f6dd4f7548f3c29: No bootstrap required, opened a new log
I20260926 11:30:21.561625 8419 ts_tablet_manager.cc:1397] T d8ae0659deb541909c73272a5fe13627 P 604a0651b0d94c288f6dd4f7548f3c29: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 11:30:21.561688 8417 ts_tablet_manager.cc:1428] T d8ae0659deb541909c73272a5fe13627 P a4a63e8498964e47b64a3d761292bd49: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260926 11:30:21.562122 8419 raft_consensus.cc:348] T d8ae0659deb541909c73272a5fe13627 P 604a0651b0d94c288f6dd4f7548f3c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.562255 8419 raft_consensus.cc:374] T d8ae0659deb541909c73272a5fe13627 P 604a0651b0d94c288f6dd4f7548f3c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:21.562299 8419 raft_consensus.cc:729] T d8ae0659deb541909c73272a5fe13627 P 604a0651b0d94c288f6dd4f7548f3c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 604a0651b0d94c288f6dd4f7548f3c29, State: Initialized, Role: FOLLOWER
I20260926 11:30:21.562427 8419 consensus_queue.cc:260] T d8ae0659deb541909c73272a5fe13627 P 604a0651b0d94c288f6dd4f7548f3c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.562706 8419 ts_tablet_manager.cc:1428] T d8ae0659deb541909c73272a5fe13627 P 604a0651b0d94c288f6dd4f7548f3c29: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 11:30:21.562656 8424 tablet_replica.cc:439] T d8ae0659deb541909c73272a5fe13627 P 604a0651b0d94c288f6dd4f7548f3c29: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } } }
I20260926 11:30:21.562790 8424 tablet_replica.cc:442] T d8ae0659deb541909c73272a5fe13627 P 604a0651b0d94c288f6dd4f7548f3c29: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:30:21.567348 8437 raft_consensus.cc:482] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:30:21.567440 8437 raft_consensus.cc:504] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.567776 8437 leader_election.cc:288] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 604a0651b0d94c288f6dd4f7548f3c29 (127.0.98.2:36991), a4a63e8498964e47b64a3d761292bd49 (127.0.98.3:32989)
I20260926 11:30:21.568126 8203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8ae0659deb541909c73272a5fe13627" candidate_uuid: "019b6a6a8c374064be3f9293e1b62077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "604a0651b0d94c288f6dd4f7548f3c29" is_pre_election: true
I20260926 11:30:21.568117 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8ae0659deb541909c73272a5fe13627" candidate_uuid: "019b6a6a8c374064be3f9293e1b62077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4a63e8498964e47b64a3d761292bd49" is_pre_election: true
I20260926 11:30:21.568270 8203 raft_consensus.cc:2454] T d8ae0659deb541909c73272a5fe13627 P 604a0651b0d94c288f6dd4f7548f3c29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 019b6a6a8c374064be3f9293e1b62077 in term 0.
I20260926 11:30:21.568269 8343 raft_consensus.cc:2454] T d8ae0659deb541909c73272a5fe13627 P a4a63e8498964e47b64a3d761292bd49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 019b6a6a8c374064be3f9293e1b62077 in term 0.
I20260926 11:30:21.568519 7994 leader_election.cc:302] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 019b6a6a8c374064be3f9293e1b62077, 604a0651b0d94c288f6dd4f7548f3c29; no voters:
I20260926 11:30:21.568710 8437 raft_consensus.cc:2781] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:30:21.568778 8437 raft_consensus.cc:482] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:30:21.568831 8437 raft_consensus.cc:3037] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:21.569769 8437 raft_consensus.cc:504] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.570101 8437 leader_election.cc:288] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077 [CANDIDATE]: Term 1 election: Requested vote from peers 604a0651b0d94c288f6dd4f7548f3c29 (127.0.98.2:36991), a4a63e8498964e47b64a3d761292bd49 (127.0.98.3:32989)
I20260926 11:30:21.570415 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8ae0659deb541909c73272a5fe13627" candidate_uuid: "019b6a6a8c374064be3f9293e1b62077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4a63e8498964e47b64a3d761292bd49"
I20260926 11:30:21.570415 8203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8ae0659deb541909c73272a5fe13627" candidate_uuid: "019b6a6a8c374064be3f9293e1b62077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "604a0651b0d94c288f6dd4f7548f3c29"
I20260926 11:30:21.570523 8343 raft_consensus.cc:3037] T d8ae0659deb541909c73272a5fe13627 P a4a63e8498964e47b64a3d761292bd49 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:21.570523 8203 raft_consensus.cc:3037] T d8ae0659deb541909c73272a5fe13627 P 604a0651b0d94c288f6dd4f7548f3c29 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:21.571569 8203 raft_consensus.cc:2454] T d8ae0659deb541909c73272a5fe13627 P 604a0651b0d94c288f6dd4f7548f3c29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 019b6a6a8c374064be3f9293e1b62077 in term 1.
I20260926 11:30:21.571569 8343 raft_consensus.cc:2454] T d8ae0659deb541909c73272a5fe13627 P a4a63e8498964e47b64a3d761292bd49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 019b6a6a8c374064be3f9293e1b62077 in term 1.
I20260926 11:30:21.571863 7994 leader_election.cc:302] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 019b6a6a8c374064be3f9293e1b62077, 604a0651b0d94c288f6dd4f7548f3c29; no voters:
I20260926 11:30:21.572074 8437 raft_consensus.cc:2781] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:30:21.572170 8437 raft_consensus.cc:686] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077 [term 1 LEADER]: Becoming Leader. State: Replica: 019b6a6a8c374064be3f9293e1b62077, State: Running, Role: LEADER
I20260926 11:30:21.572317 8437 consensus_queue.cc:237] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.573010 8437 tablet_replica.cc:439] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077: TxnStatusTablet state changed. Reason: New leader 019b6a6a8c374064be3f9293e1b62077. Latest consensus state: current_term: 1 leader_uuid: "019b6a6a8c374064be3f9293e1b62077" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } } }
I20260926 11:30:21.573101 8437 tablet_replica.cc:442] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077: This TxnStatusTablet replica's current role is: LEADER
I20260926 11:30:21.573338 8459 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 11:30:21.573534 8459 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 11:30:21.573932 7905 catalog_manager.cc:5495] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077 reported cstate change: term changed from 0 to 1, leader changed from <none> to 019b6a6a8c374064be3f9293e1b62077 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "019b6a6a8c374064be3f9293e1b62077" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } health_report { overall_health: UNKNOWN } } }
I20260926 11:30:21.585983 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:21.582761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60984 (local address 127.0.98.1:41563)
0926 11:30:21.582882 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:21.582885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:21.582899 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:30:21.583081 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:21.583084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:21.583134 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:21.583264 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:30:21.583520 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.584084 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.584939 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.585148 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.585260 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:21.585526 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:30:21.585602 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:30:21.585675 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:21.585789 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 11:30:21.585838 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 11:30:21.585983 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:21.582672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41563 (local address 127.0.0.1:60984)
0926 11:30:21.582819 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:21.582837 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:21.582934 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:21.583290 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:21.583293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:21.583301 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 11:30:21.583411 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:21.583416 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.584196 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.584200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:21.584819 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:21.584826 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.585255 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.585259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:21.585347 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:21.585395 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 11:30:21.585690 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 11:30:21.585713 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 11:30:21.585776 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 11:30:21.585860 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 11:30:21.588548 8343 raft_consensus.cc:1261] T d8ae0659deb541909c73272a5fe13627 P a4a63e8498964e47b64a3d761292bd49 [term 1 FOLLOWER]: Refusing update from remote peer 019b6a6a8c374064be3f9293e1b62077: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:30:21.588953 8425 consensus_queue.cc:1035] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:30:21.589003 8203 raft_consensus.cc:1261] T d8ae0659deb541909c73272a5fe13627 P 604a0651b0d94c288f6dd4f7548f3c29 [term 1 FOLLOWER]: Refusing update from remote peer 019b6a6a8c374064be3f9293e1b62077: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:30:21.589453 8425 consensus_queue.cc:1035] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:30:21.590535 8423 tablet_replica.cc:439] T d8ae0659deb541909c73272a5fe13627 P a4a63e8498964e47b64a3d761292bd49: TxnStatusTablet state changed. Reason: New leader 019b6a6a8c374064be3f9293e1b62077. Latest consensus state: current_term: 1 leader_uuid: "019b6a6a8c374064be3f9293e1b62077" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } } }
I20260926 11:30:21.590653 8423 tablet_replica.cc:442] T d8ae0659deb541909c73272a5fe13627 P a4a63e8498964e47b64a3d761292bd49: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:30:21.590966 8424 tablet_replica.cc:439] T d8ae0659deb541909c73272a5fe13627 P 604a0651b0d94c288f6dd4f7548f3c29: TxnStatusTablet state changed. Reason: New leader 019b6a6a8c374064be3f9293e1b62077. Latest consensus state: current_term: 1 leader_uuid: "019b6a6a8c374064be3f9293e1b62077" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } } }
I20260926 11:30:21.591073 8424 tablet_replica.cc:442] T d8ae0659deb541909c73272a5fe13627 P 604a0651b0d94c288f6dd4f7548f3c29: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:30:21.592279 8428 tablet_replica.cc:439] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "019b6a6a8c374064be3f9293e1b62077" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } } }
I20260926 11:30:21.592332 8431 mvcc.cc:205] Tried to move back new op lower bound from 7333569419622182912 to 7333569419561680896. Current Snapshot: MvccSnapshot[applied={T|T < 7333569419622182912}]
I20260926 11:30:21.592386 8428 tablet_replica.cc:442] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077: This TxnStatusTablet replica's current role is: LEADER
I20260926 11:30:21.593924 8423 tablet_replica.cc:439] T d8ae0659deb541909c73272a5fe13627 P a4a63e8498964e47b64a3d761292bd49: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "019b6a6a8c374064be3f9293e1b62077" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } } }
I20260926 11:30:21.594026 8423 tablet_replica.cc:442] T d8ae0659deb541909c73272a5fe13627 P a4a63e8498964e47b64a3d761292bd49: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:30:21.593987 8432 mvcc.cc:205] Tried to move back new op lower bound from 7333569419622182912 to 7333569419561680896. Current Snapshot: MvccSnapshot[applied={T|T < 7333569419622182912}]
I20260926 11:30:21.597146 8428 tablet_replica.cc:439] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "019b6a6a8c374064be3f9293e1b62077" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } } }
I20260926 11:30:21.597260 8428 tablet_replica.cc:442] T d8ae0659deb541909c73272a5fe13627 P 019b6a6a8c374064be3f9293e1b62077: This TxnStatusTablet replica's current role is: LEADER
I20260926 11:30:21.598073 8424 tablet_replica.cc:439] T d8ae0659deb541909c73272a5fe13627 P 604a0651b0d94c288f6dd4f7548f3c29: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "019b6a6a8c374064be3f9293e1b62077" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } } }
I20260926 11:30:21.598176 8424 tablet_replica.cc:442] T d8ae0659deb541909c73272a5fe13627 P 604a0651b0d94c288f6dd4f7548f3c29: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 11:30:21.607198 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:21.603809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56831 (local address 127.0.98.1:41563)
0926 11:30:21.603936 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:21.603941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:21.604164 (+ 223us) server_negotiation.cc:407] Connection header received
0926 11:30:21.604273 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:21.604276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:21.604317 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:21.604384 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:30:21.604591 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.605070 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.606231 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.606428 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.606498 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:21.606785 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:30:21.606848 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:30:21.606899 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:21.607004 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 11:30:21.607076 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":48}
I20260926 11:30:21.607297 8466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:21.603726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41563 (local address 127.0.98.1:56831)
0926 11:30:21.604054 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:21.604069 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:21.604177 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:21.604405 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:21.604407 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:21.604415 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 11:30:21.604515 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:21.604520 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.605158 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.605161 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:21.606113 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:21.606123 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.606519 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.606522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:21.606605 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:21.606656 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 11:30:21.606921 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 11:30:21.606927 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 11:30:21.606997 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 11:30:21.607117 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":21,"thread_start_us":125,"threads_started":1}
I20260926 11:30:21.613509 8343 raft_consensus.cc:1261] T 20d9e0ed6bfe4f508646fb170a17996c P a4a63e8498964e47b64a3d761292bd49 [term 1 FOLLOWER]: Refusing update from remote peer 019b6a6a8c374064be3f9293e1b62077: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:30:21.613687 8203 raft_consensus.cc:1261] T 20d9e0ed6bfe4f508646fb170a17996c P 604a0651b0d94c288f6dd4f7548f3c29 [term 1 FOLLOWER]: Refusing update from remote peer 019b6a6a8c374064be3f9293e1b62077: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:30:21.613945 8425 consensus_queue.cc:1035] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:30:21.614187 8428 consensus_queue.cc:1035] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:30:21.658349 7906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:30:21.658485 7906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34270:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:30:21.660475 7906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c53348dae04456b9c59629c03d412d6: Sending DeleteTablet for 3 replicas of tablet 20d9e0ed6bfe4f508646fb170a17996c
I20260926 11:30:21.661015 8165 tablet_service.cc:1514] Processing DeleteTablet for tablet 20d9e0ed6bfe4f508646fb170a17996c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:30:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35382
I20260926 11:30:21.661015 8043 tablet_service.cc:1514] Processing DeleteTablet for tablet 20d9e0ed6bfe4f508646fb170a17996c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:30:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60964
I20260926 11:30:21.661072 8323 tablet_service.cc:1514] Processing DeleteTablet for tablet 20d9e0ed6bfe4f508646fb170a17996c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:30:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46356
I20260926 11:30:21.661216 8441 tablet_replica.cc:330] T 20d9e0ed6bfe4f508646fb170a17996c P 604a0651b0d94c288f6dd4f7548f3c29: stopping tablet replica
I20260926 11:30:21.661363 8441 raft_consensus.cc:2229] T 20d9e0ed6bfe4f508646fb170a17996c P 604a0651b0d94c288f6dd4f7548f3c29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:30:21.661516 8441 raft_consensus.cc:2258] T 20d9e0ed6bfe4f508646fb170a17996c P 604a0651b0d94c288f6dd4f7548f3c29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:30:21.661707 8443 tablet_replica.cc:330] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077: stopping tablet replica
I20260926 11:30:21.661834 8443 raft_consensus.cc:2229] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:30:21.662019 8443 raft_consensus.cc:2258] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:30:21.662230 8442 tablet_replica.cc:330] T 20d9e0ed6bfe4f508646fb170a17996c P a4a63e8498964e47b64a3d761292bd49: stopping tablet replica
I20260926 11:30:21.662237 8441 ts_tablet_manager.cc:1905] T 20d9e0ed6bfe4f508646fb170a17996c P 604a0651b0d94c288f6dd4f7548f3c29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:30:21.662359 8442 raft_consensus.cc:2229] T 20d9e0ed6bfe4f508646fb170a17996c P a4a63e8498964e47b64a3d761292bd49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:30:21.662478 8442 raft_consensus.cc:2258] T 20d9e0ed6bfe4f508646fb170a17996c P a4a63e8498964e47b64a3d761292bd49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:30:21.663146 8443 ts_tablet_manager.cc:1905] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:30:21.663919 8442 ts_tablet_manager.cc:1905] T 20d9e0ed6bfe4f508646fb170a17996c P a4a63e8498964e47b64a3d761292bd49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:30:21.665143 8441 ts_tablet_manager.cc:1918] T 20d9e0ed6bfe4f508646fb170a17996c P 604a0651b0d94c288f6dd4f7548f3c29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 11:30:21.665220 8441 log.cc:1196] T 20d9e0ed6bfe4f508646fb170a17996c P 604a0651b0d94c288f6dd4f7548f3c29: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-1/wal/wals/20d9e0ed6bfe4f508646fb170a17996c
I20260926 11:30:21.665488 8441 ts_tablet_manager.cc:1939] T 20d9e0ed6bfe4f508646fb170a17996c P 604a0651b0d94c288f6dd4f7548f3c29: Deleting consensus metadata
I20260926 11:30:21.665889 8443 ts_tablet_manager.cc:1918] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 11:30:21.665949 8443 log.cc:1196] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-0/wal/wals/20d9e0ed6bfe4f508646fb170a17996c
I20260926 11:30:21.666182 8443 ts_tablet_manager.cc:1939] T 20d9e0ed6bfe4f508646fb170a17996c P 019b6a6a8c374064be3f9293e1b62077: Deleting consensus metadata
I20260926 11:30:21.666767 8442 ts_tablet_manager.cc:1918] T 20d9e0ed6bfe4f508646fb170a17996c P a4a63e8498964e47b64a3d761292bd49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 11:30:21.666834 8442 log.cc:1196] T 20d9e0ed6bfe4f508646fb170a17996c P a4a63e8498964e47b64a3d761292bd49: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-2/wal/wals/20d9e0ed6bfe4f508646fb170a17996c
I20260926 11:30:21.666977 7891 catalog_manager.cc:4841] TS 019b6a6a8c374064be3f9293e1b62077 (127.0.98.1:41563): tablet 20d9e0ed6bfe4f508646fb170a17996c (table test-table [id=1e22bbe924a64e9c9722cd0326b04f70]) successfully deleted
I20260926 11:30:21.667066 8442 ts_tablet_manager.cc:1939] T 20d9e0ed6bfe4f508646fb170a17996c P a4a63e8498964e47b64a3d761292bd49: Deleting consensus metadata
I20260926 11:30:21.667711 7893 catalog_manager.cc:4841] TS a4a63e8498964e47b64a3d761292bd49 (127.0.98.3:32989): tablet 20d9e0ed6bfe4f508646fb170a17996c (table test-table [id=1e22bbe924a64e9c9722cd0326b04f70]) successfully deleted
I20260926 11:30:21.667765 7890 catalog_manager.cc:4841] TS 604a0651b0d94c288f6dd4f7548f3c29 (127.0.98.2:36991): tablet 20d9e0ed6bfe4f508646fb170a17996c (table test-table [id=1e22bbe924a64e9c9722cd0326b04f70]) successfully deleted
Sep 26 11:30:21 dist-test-slave-mrck krb5kdc[7861](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 11:30:21 dist-test-slave-mrck krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:21 dist-test-slave-mrck krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 11:30:21.695894 8454 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:21.688878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34288 (local address 127.0.98.62:36439)
0926 11:30:21.689013 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:21.689019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:21.689217 (+ 198us) server_negotiation.cc:407] Connection header received
0926 11:30:21.689383 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:21.689386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:21.689440 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:21.689529 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:21.690246 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.690774 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.691634 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.691956 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.692066 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:21.695762 (+ 3696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34288: BlockingRecv error: recv got EOF from 127.0.0.1:34288 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 11:30:21 dist-test-slave-mrck krb5kdc[7861](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 11:30:21.706372 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:21.697568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34302 (local address 127.0.98.62:36439)
0926 11:30:21.697693 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:21.697717 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:21.697838 (+ 121us) server_negotiation.cc:407] Connection header received
0926 11:30:21.697976 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:21.697978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:21.698018 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:21.698083 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:21.698847 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.699407 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.700545 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.700751 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.700830 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:21.703308 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:21.703317 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:21.703319 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:21.703343 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:21.704785 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:21.705199 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:21.705204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:21.705206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:21.705282 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:21.705545 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:21.705550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:21.705551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:21.705723 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:21.705859 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:21.706086 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 11:30:21.706204 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 11:30:21.708983 7906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:30:21.709478 7906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:30:21.714322 8043 tablet_service.cc:1467] Processing CreateTablet for tablet d6970383d3b542b085292cef9af4fc5f (DEFAULT_TABLE table=test-table [id=1da17c8329414e12adff8edafb4fd3a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:30:21.714509 8165 tablet_service.cc:1467] Processing CreateTablet for tablet d6970383d3b542b085292cef9af4fc5f (DEFAULT_TABLE table=test-table [id=1da17c8329414e12adff8edafb4fd3a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:30:21.714529 8323 tablet_service.cc:1467] Processing CreateTablet for tablet d6970383d3b542b085292cef9af4fc5f (DEFAULT_TABLE table=test-table [id=1da17c8329414e12adff8edafb4fd3a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:30:21.714663 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6970383d3b542b085292cef9af4fc5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:21.714749 8165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6970383d3b542b085292cef9af4fc5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:21.714825 8323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6970383d3b542b085292cef9af4fc5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:21.717340 8419 tablet_bootstrap.cc:492] T d6970383d3b542b085292cef9af4fc5f P 604a0651b0d94c288f6dd4f7548f3c29: Bootstrap starting.
I20260926 11:30:21.718235 8419 tablet_bootstrap.cc:654] T d6970383d3b542b085292cef9af4fc5f P 604a0651b0d94c288f6dd4f7548f3c29: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:21.718827 8418 tablet_bootstrap.cc:492] T d6970383d3b542b085292cef9af4fc5f P 019b6a6a8c374064be3f9293e1b62077: Bootstrap starting.
I20260926 11:30:21.719419 8419 tablet_bootstrap.cc:492] T d6970383d3b542b085292cef9af4fc5f P 604a0651b0d94c288f6dd4f7548f3c29: No bootstrap required, opened a new log
I20260926 11:30:21.719519 8419 ts_tablet_manager.cc:1397] T d6970383d3b542b085292cef9af4fc5f P 604a0651b0d94c288f6dd4f7548f3c29: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 11:30:21.719609 8418 tablet_bootstrap.cc:654] T d6970383d3b542b085292cef9af4fc5f P 019b6a6a8c374064be3f9293e1b62077: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:21.720053 8419 raft_consensus.cc:348] T d6970383d3b542b085292cef9af4fc5f P 604a0651b0d94c288f6dd4f7548f3c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.720173 8419 raft_consensus.cc:374] T d6970383d3b542b085292cef9af4fc5f P 604a0651b0d94c288f6dd4f7548f3c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:21.720201 8417 tablet_bootstrap.cc:492] T d6970383d3b542b085292cef9af4fc5f P a4a63e8498964e47b64a3d761292bd49: Bootstrap starting.
I20260926 11:30:21.720207 8419 raft_consensus.cc:729] T d6970383d3b542b085292cef9af4fc5f P 604a0651b0d94c288f6dd4f7548f3c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 604a0651b0d94c288f6dd4f7548f3c29, State: Initialized, Role: FOLLOWER
I20260926 11:30:21.720333 8419 consensus_queue.cc:260] T d6970383d3b542b085292cef9af4fc5f P 604a0651b0d94c288f6dd4f7548f3c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.720628 8419 ts_tablet_manager.cc:1428] T d6970383d3b542b085292cef9af4fc5f P 604a0651b0d94c288f6dd4f7548f3c29: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 11:30:21.721019 8417 tablet_bootstrap.cc:654] T d6970383d3b542b085292cef9af4fc5f P a4a63e8498964e47b64a3d761292bd49: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:21.722170 8417 tablet_bootstrap.cc:492] T d6970383d3b542b085292cef9af4fc5f P a4a63e8498964e47b64a3d761292bd49: No bootstrap required, opened a new log
I20260926 11:30:21.722252 8417 ts_tablet_manager.cc:1397] T d6970383d3b542b085292cef9af4fc5f P a4a63e8498964e47b64a3d761292bd49: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 11:30:21.722664 8418 tablet_bootstrap.cc:492] T d6970383d3b542b085292cef9af4fc5f P 019b6a6a8c374064be3f9293e1b62077: No bootstrap required, opened a new log
I20260926 11:30:21.722745 8418 ts_tablet_manager.cc:1397] T d6970383d3b542b085292cef9af4fc5f P 019b6a6a8c374064be3f9293e1b62077: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260926 11:30:21.722747 8417 raft_consensus.cc:348] T d6970383d3b542b085292cef9af4fc5f P a4a63e8498964e47b64a3d761292bd49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.722878 8417 raft_consensus.cc:374] T d6970383d3b542b085292cef9af4fc5f P a4a63e8498964e47b64a3d761292bd49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:21.722919 8417 raft_consensus.cc:729] T d6970383d3b542b085292cef9af4fc5f P a4a63e8498964e47b64a3d761292bd49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4a63e8498964e47b64a3d761292bd49, State: Initialized, Role: FOLLOWER
I20260926 11:30:21.723048 8417 consensus_queue.cc:260] T d6970383d3b542b085292cef9af4fc5f P a4a63e8498964e47b64a3d761292bd49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.723145 8418 raft_consensus.cc:348] T d6970383d3b542b085292cef9af4fc5f P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.723270 8418 raft_consensus.cc:374] T d6970383d3b542b085292cef9af4fc5f P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:21.723317 8418 raft_consensus.cc:729] T d6970383d3b542b085292cef9af4fc5f P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 019b6a6a8c374064be3f9293e1b62077, State: Initialized, Role: FOLLOWER
I20260926 11:30:21.723377 8417 ts_tablet_manager.cc:1428] T d6970383d3b542b085292cef9af4fc5f P a4a63e8498964e47b64a3d761292bd49: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 11:30:21.723435 8418 consensus_queue.cc:260] T d6970383d3b542b085292cef9af4fc5f P 019b6a6a8c374064be3f9293e1b62077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.723702 8418 ts_tablet_manager.cc:1428] T d6970383d3b542b085292cef9af4fc5f P 019b6a6a8c374064be3f9293e1b62077: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 11:30:21.834102 8423 raft_consensus.cc:482] T d6970383d3b542b085292cef9af4fc5f P a4a63e8498964e47b64a3d761292bd49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:30:21.834280 8423 raft_consensus.cc:504] T d6970383d3b542b085292cef9af4fc5f P a4a63e8498964e47b64a3d761292bd49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.835465 8423 leader_election.cc:288] T d6970383d3b542b085292cef9af4fc5f P a4a63e8498964e47b64a3d761292bd49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 019b6a6a8c374064be3f9293e1b62077 (127.0.98.1:41563), 604a0651b0d94c288f6dd4f7548f3c29 (127.0.98.2:36991)
I20260926 11:30:21.839432 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:21.835895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34105 (local address 127.0.98.1:41563)
0926 11:30:21.836030 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:21.836035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:21.836201 (+ 166us) server_negotiation.cc:407] Connection header received
0926 11:30:21.836366 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:21.836369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:21.836423 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:21.836506 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:30:21.836777 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.837289 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.838451 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.838861 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.838979 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:21.839046 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:21.839224 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 11:30:21.839292 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 11:30:21.839464 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:21.835767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41563 (local address 127.0.98.3:34105)
0926 11:30:21.836101 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:21.836113 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:21.836260 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:21.836518 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:21.836521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:21.836536 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:30:21.836659 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:21.836664 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.837408 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.837411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:21.838329 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:21.838338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.838987 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.838990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:21.839088 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:21.839110 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 11:30:21.839208 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 11:30:21.839286 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":21,"thread_start_us":89,"threads_started":1}
I20260926 11:30:21.839887 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:21.835890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32779 (local address 127.0.98.2:36991)
0926 11:30:21.836191 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:21.836196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:21.836356 (+ 160us) server_negotiation.cc:407] Connection header received
0926 11:30:21.836505 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:21.836508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:21.836554 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:21.836640 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:30:21.836912 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.837420 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.838732 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.839132 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.839249 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:21.839317 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:21.839591 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 11:30:21.839689 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":73,"threads_started":1}
I20260926 11:30:21.840050 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6970383d3b542b085292cef9af4fc5f" candidate_uuid: "a4a63e8498964e47b64a3d761292bd49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "019b6a6a8c374064be3f9293e1b62077" is_pre_election: true
I20260926 11:30:21.840266 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:21.835785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36991 (local address 127.0.98.3:32779)
0926 11:30:21.836265 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:21.836277 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:21.836396 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:21.836659 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:21.836662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:21.836678 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:30:21.836804 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:21.836809 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.837824 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.837826 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:21.838597 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:21.838606 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.839283 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.839287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:21.839432 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:21.839454 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 11:30:21.840037 (+ 583us) client_negotiation.cc:240] Negotiation successful
0926 11:30:21.840119 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":23,"thread_start_us":79,"threads_started":1}
I20260926 11:30:21.840344 8063 raft_consensus.cc:2454] T d6970383d3b542b085292cef9af4fc5f P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4a63e8498964e47b64a3d761292bd49 in term 0.
I20260926 11:30:21.840852 8203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6970383d3b542b085292cef9af4fc5f" candidate_uuid: "a4a63e8498964e47b64a3d761292bd49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "604a0651b0d94c288f6dd4f7548f3c29" is_pre_election: true
I20260926 11:30:21.840830 8275 leader_election.cc:302] T d6970383d3b542b085292cef9af4fc5f P a4a63e8498964e47b64a3d761292bd49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 019b6a6a8c374064be3f9293e1b62077, a4a63e8498964e47b64a3d761292bd49; no voters:
I20260926 11:30:21.841013 8203 raft_consensus.cc:2454] T d6970383d3b542b085292cef9af4fc5f P 604a0651b0d94c288f6dd4f7548f3c29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4a63e8498964e47b64a3d761292bd49 in term 0.
I20260926 11:30:21.841182 8423 raft_consensus.cc:2781] T d6970383d3b542b085292cef9af4fc5f P a4a63e8498964e47b64a3d761292bd49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:30:21.841253 8423 raft_consensus.cc:482] T d6970383d3b542b085292cef9af4fc5f P a4a63e8498964e47b64a3d761292bd49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:30:21.841297 8423 raft_consensus.cc:3037] T d6970383d3b542b085292cef9af4fc5f P a4a63e8498964e47b64a3d761292bd49 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:21.842317 8423 raft_consensus.cc:504] T d6970383d3b542b085292cef9af4fc5f P a4a63e8498964e47b64a3d761292bd49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.842650 8423 leader_election.cc:288] T d6970383d3b542b085292cef9af4fc5f P a4a63e8498964e47b64a3d761292bd49 [CANDIDATE]: Term 1 election: Requested vote from peers 019b6a6a8c374064be3f9293e1b62077 (127.0.98.1:41563), 604a0651b0d94c288f6dd4f7548f3c29 (127.0.98.2:36991)
I20260926 11:30:21.842998 8063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6970383d3b542b085292cef9af4fc5f" candidate_uuid: "a4a63e8498964e47b64a3d761292bd49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "019b6a6a8c374064be3f9293e1b62077"
I20260926 11:30:21.843017 8203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6970383d3b542b085292cef9af4fc5f" candidate_uuid: "a4a63e8498964e47b64a3d761292bd49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "604a0651b0d94c288f6dd4f7548f3c29"
I20260926 11:30:21.843132 8063 raft_consensus.cc:3037] T d6970383d3b542b085292cef9af4fc5f P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:21.843142 8203 raft_consensus.cc:3037] T d6970383d3b542b085292cef9af4fc5f P 604a0651b0d94c288f6dd4f7548f3c29 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:21.844146 8203 raft_consensus.cc:2454] T d6970383d3b542b085292cef9af4fc5f P 604a0651b0d94c288f6dd4f7548f3c29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4a63e8498964e47b64a3d761292bd49 in term 1.
I20260926 11:30:21.844162 8063 raft_consensus.cc:2454] T d6970383d3b542b085292cef9af4fc5f P 019b6a6a8c374064be3f9293e1b62077 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4a63e8498964e47b64a3d761292bd49 in term 1.
I20260926 11:30:21.844444 8274 leader_election.cc:302] T d6970383d3b542b085292cef9af4fc5f P a4a63e8498964e47b64a3d761292bd49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 604a0651b0d94c288f6dd4f7548f3c29, a4a63e8498964e47b64a3d761292bd49; no voters:
I20260926 11:30:21.844628 8423 raft_consensus.cc:2781] T d6970383d3b542b085292cef9af4fc5f P a4a63e8498964e47b64a3d761292bd49 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:30:21.844815 8423 raft_consensus.cc:686] T d6970383d3b542b085292cef9af4fc5f P a4a63e8498964e47b64a3d761292bd49 [term 1 LEADER]: Becoming Leader. State: Replica: a4a63e8498964e47b64a3d761292bd49, State: Running, Role: LEADER
I20260926 11:30:21.845062 8423 consensus_queue.cc:237] T d6970383d3b542b085292cef9af4fc5f P a4a63e8498964e47b64a3d761292bd49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.847587 7903 catalog_manager.cc:5495] T d6970383d3b542b085292cef9af4fc5f P a4a63e8498964e47b64a3d761292bd49 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4a63e8498964e47b64a3d761292bd49 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a4a63e8498964e47b64a3d761292bd49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } health_report { overall_health: HEALTHY } } }
I20260926 11:30:21.877871 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:21.874220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46364 (local address 127.0.98.3:32989)
0926 11:30:21.874547 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:21.874554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:21.874568 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:30:21.874645 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:21.874647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:21.874704 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:21.874816 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:30:21.875042 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.875558 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.876466 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:21.876688 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:21.876774 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:21.877004 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:30:21.877456 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:30:21.877545 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:21.877597 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 11:30:21.877641 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":106,"threads_started":1}
I20260926 11:30:21.880601 8203 raft_consensus.cc:1261] T d6970383d3b542b085292cef9af4fc5f P 604a0651b0d94c288f6dd4f7548f3c29 [term 1 FOLLOWER]: Refusing update from remote peer a4a63e8498964e47b64a3d761292bd49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:30:21.880884 8063 raft_consensus.cc:1261] T d6970383d3b542b085292cef9af4fc5f P 019b6a6a8c374064be3f9293e1b62077 [term 1 FOLLOWER]: Refusing update from remote peer a4a63e8498964e47b64a3d761292bd49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:30:21.881167 8423 consensus_queue.cc:1035] T d6970383d3b542b085292cef9af4fc5f P a4a63e8498964e47b64a3d761292bd49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:30:21.881501 8488 consensus_queue.cc:1035] T d6970383d3b542b085292cef9af4fc5f P a4a63e8498964e47b64a3d761292bd49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:30:21.891841 7906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:30:21.891975 7906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:30:21.894048 7906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c53348dae04456b9c59629c03d412d6: Sending DeleteTablet for 3 replicas of tablet d6970383d3b542b085292cef9af4fc5f
I20260926 11:30:21.894569 8043 tablet_service.cc:1514] Processing DeleteTablet for tablet d6970383d3b542b085292cef9af4fc5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:30:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60964
I20260926 11:30:21.894685 8323 tablet_service.cc:1514] Processing DeleteTablet for tablet d6970383d3b542b085292cef9af4fc5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:30:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46356
I20260926 11:30:21.894776 8443 tablet_replica.cc:330] T d6970383d3b542b085292cef9af4fc5f P 019b6a6a8c374064be3f9293e1b62077: stopping tablet replica
I20260926 11:30:21.894851 8442 tablet_replica.cc:330] T d6970383d3b542b085292cef9af4fc5f P a4a63e8498964e47b64a3d761292bd49: stopping tablet replica
I20260926 11:30:21.894910 8443 raft_consensus.cc:2229] T d6970383d3b542b085292cef9af4fc5f P 019b6a6a8c374064be3f9293e1b62077 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:30:21.894943 8442 raft_consensus.cc:2229] T d6970383d3b542b085292cef9af4fc5f P a4a63e8498964e47b64a3d761292bd49 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:30:21.895040 8443 raft_consensus.cc:2258] T d6970383d3b542b085292cef9af4fc5f P 019b6a6a8c374064be3f9293e1b62077 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:30:21.895180 8442 raft_consensus.cc:2258] T d6970383d3b542b085292cef9af4fc5f P a4a63e8498964e47b64a3d761292bd49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:30:21.895596 8165 tablet_service.cc:1514] Processing DeleteTablet for tablet d6970383d3b542b085292cef9af4fc5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:30:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35382
I20260926 11:30:21.895843 8441 tablet_replica.cc:330] T d6970383d3b542b085292cef9af4fc5f P 604a0651b0d94c288f6dd4f7548f3c29: stopping tablet replica
I20260926 11:30:21.895859 8442 ts_tablet_manager.cc:1905] T d6970383d3b542b085292cef9af4fc5f P a4a63e8498964e47b64a3d761292bd49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:30:21.895954 8441 raft_consensus.cc:2229] T d6970383d3b542b085292cef9af4fc5f P 604a0651b0d94c288f6dd4f7548f3c29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:30:21.896077 8441 raft_consensus.cc:2258] T d6970383d3b542b085292cef9af4fc5f P 604a0651b0d94c288f6dd4f7548f3c29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:30:21.896200 7906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:30:21.896567 7906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:30:21.896971 8443 ts_tablet_manager.cc:1905] T d6970383d3b542b085292cef9af4fc5f P 019b6a6a8c374064be3f9293e1b62077: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:30:21.897325 8441 ts_tablet_manager.cc:1905] T d6970383d3b542b085292cef9af4fc5f P 604a0651b0d94c288f6dd4f7548f3c29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:30:21.898336 8442 ts_tablet_manager.cc:1918] T d6970383d3b542b085292cef9af4fc5f P a4a63e8498964e47b64a3d761292bd49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:30:21.898392 8442 log.cc:1196] T d6970383d3b542b085292cef9af4fc5f P a4a63e8498964e47b64a3d761292bd49: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-2/wal/wals/d6970383d3b542b085292cef9af4fc5f
I20260926 11:30:21.898574 8442 ts_tablet_manager.cc:1939] T d6970383d3b542b085292cef9af4fc5f P a4a63e8498964e47b64a3d761292bd49: Deleting consensus metadata
I20260926 11:30:21.899353 7893 catalog_manager.cc:4841] TS a4a63e8498964e47b64a3d761292bd49 (127.0.98.3:32989): tablet d6970383d3b542b085292cef9af4fc5f (table test-table [id=1da17c8329414e12adff8edafb4fd3a9]) successfully deleted
I20260926 11:30:21.899600 8443 ts_tablet_manager.cc:1918] T d6970383d3b542b085292cef9af4fc5f P 019b6a6a8c374064be3f9293e1b62077: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:30:21.899680 8443 log.cc:1196] T d6970383d3b542b085292cef9af4fc5f P 019b6a6a8c374064be3f9293e1b62077: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-0/wal/wals/d6970383d3b542b085292cef9af4fc5f
I20260926 11:30:21.900032 8443 ts_tablet_manager.cc:1939] T d6970383d3b542b085292cef9af4fc5f P 019b6a6a8c374064be3f9293e1b62077: Deleting consensus metadata
I20260926 11:30:21.900553 8441 ts_tablet_manager.cc:1918] T d6970383d3b542b085292cef9af4fc5f P 604a0651b0d94c288f6dd4f7548f3c29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:30:21.900658 8441 log.cc:1196] T d6970383d3b542b085292cef9af4fc5f P 604a0651b0d94c288f6dd4f7548f3c29: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-1/wal/wals/d6970383d3b542b085292cef9af4fc5f
I20260926 11:30:21.900806 7891 catalog_manager.cc:4841] TS 019b6a6a8c374064be3f9293e1b62077 (127.0.98.1:41563): tablet d6970383d3b542b085292cef9af4fc5f (table test-table [id=1da17c8329414e12adff8edafb4fd3a9]) successfully deleted
I20260926 11:30:21.900925 8441 ts_tablet_manager.cc:1939] T d6970383d3b542b085292cef9af4fc5f P 604a0651b0d94c288f6dd4f7548f3c29: Deleting consensus metadata
I20260926 11:30:21.901533 7890 catalog_manager.cc:4841] TS 604a0651b0d94c288f6dd4f7548f3c29 (127.0.98.2:36991): tablet d6970383d3b542b085292cef9af4fc5f (table test-table [id=1da17c8329414e12adff8edafb4fd3a9]) successfully deleted
I20260926 11:30:21.904198 8043 tablet_service.cc:1467] Processing CreateTablet for tablet d1569c32127d4920b49c2820c034f72b (DEFAULT_TABLE table=test-table [id=d2528f88a93442ea977d8635f0cca855]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:30:21.904263 8165 tablet_service.cc:1467] Processing CreateTablet for tablet d1569c32127d4920b49c2820c034f72b (DEFAULT_TABLE table=test-table [id=d2528f88a93442ea977d8635f0cca855]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:30:21.904491 8043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1569c32127d4920b49c2820c034f72b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:21.904510 8165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1569c32127d4920b49c2820c034f72b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:21.904575 8323 tablet_service.cc:1467] Processing CreateTablet for tablet d1569c32127d4920b49c2820c034f72b (DEFAULT_TABLE table=test-table [id=d2528f88a93442ea977d8635f0cca855]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:30:21.904820 8323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1569c32127d4920b49c2820c034f72b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:21.906980 8419 tablet_bootstrap.cc:492] T d1569c32127d4920b49c2820c034f72b P 604a0651b0d94c288f6dd4f7548f3c29: Bootstrap starting.
I20260926 11:30:21.907119 8417 tablet_bootstrap.cc:492] T d1569c32127d4920b49c2820c034f72b P a4a63e8498964e47b64a3d761292bd49: Bootstrap starting.
I20260926 11:30:21.907898 8418 tablet_bootstrap.cc:492] T d1569c32127d4920b49c2820c034f72b P 019b6a6a8c374064be3f9293e1b62077: Bootstrap starting.
I20260926 11:30:21.908051 8419 tablet_bootstrap.cc:654] T d1569c32127d4920b49c2820c034f72b P 604a0651b0d94c288f6dd4f7548f3c29: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:21.908077 8417 tablet_bootstrap.cc:654] T d1569c32127d4920b49c2820c034f72b P a4a63e8498964e47b64a3d761292bd49: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:21.908648 8418 tablet_bootstrap.cc:654] T d1569c32127d4920b49c2820c034f72b P 019b6a6a8c374064be3f9293e1b62077: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:21.909302 8419 tablet_bootstrap.cc:492] T d1569c32127d4920b49c2820c034f72b P 604a0651b0d94c288f6dd4f7548f3c29: No bootstrap required, opened a new log
I20260926 11:30:21.909408 8419 ts_tablet_manager.cc:1397] T d1569c32127d4920b49c2820c034f72b P 604a0651b0d94c288f6dd4f7548f3c29: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 11:30:21.910030 8417 tablet_bootstrap.cc:492] T d1569c32127d4920b49c2820c034f72b P a4a63e8498964e47b64a3d761292bd49: No bootstrap required, opened a new log
I20260926 11:30:21.909965 8419 raft_consensus.cc:348] T d1569c32127d4920b49c2820c034f72b P 604a0651b0d94c288f6dd4f7548f3c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.910092 8419 raft_consensus.cc:374] T d1569c32127d4920b49c2820c034f72b P 604a0651b0d94c288f6dd4f7548f3c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:21.910107 8417 ts_tablet_manager.cc:1397] T d1569c32127d4920b49c2820c034f72b P a4a63e8498964e47b64a3d761292bd49: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 11:30:21.910131 8419 raft_consensus.cc:729] T d1569c32127d4920b49c2820c034f72b P 604a0651b0d94c288f6dd4f7548f3c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 604a0651b0d94c288f6dd4f7548f3c29, State: Initialized, Role: FOLLOWER
I20260926 11:30:21.910274 8419 consensus_queue.cc:260] T d1569c32127d4920b49c2820c034f72b P 604a0651b0d94c288f6dd4f7548f3c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.910432 8418 tablet_bootstrap.cc:492] T d1569c32127d4920b49c2820c034f72b P 019b6a6a8c374064be3f9293e1b62077: No bootstrap required, opened a new log
I20260926 11:30:21.910506 8418 ts_tablet_manager.cc:1397] T d1569c32127d4920b49c2820c034f72b P 019b6a6a8c374064be3f9293e1b62077: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 11:30:21.910488 8417 raft_consensus.cc:348] T d1569c32127d4920b49c2820c034f72b P a4a63e8498964e47b64a3d761292bd49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.910576 8419 ts_tablet_manager.cc:1428] T d1569c32127d4920b49c2820c034f72b P 604a0651b0d94c288f6dd4f7548f3c29: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 11:30:21.910583 8417 raft_consensus.cc:374] T d1569c32127d4920b49c2820c034f72b P a4a63e8498964e47b64a3d761292bd49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:21.910626 8417 raft_consensus.cc:729] T d1569c32127d4920b49c2820c034f72b P a4a63e8498964e47b64a3d761292bd49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4a63e8498964e47b64a3d761292bd49, State: Initialized, Role: FOLLOWER
I20260926 11:30:21.910759 8417 consensus_queue.cc:260] T d1569c32127d4920b49c2820c034f72b P a4a63e8498964e47b64a3d761292bd49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.910995 8417 ts_tablet_manager.cc:1428] T d1569c32127d4920b49c2820c034f72b P a4a63e8498964e47b64a3d761292bd49: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 11:30:21.910960 8418 raft_consensus.cc:348] T d1569c32127d4920b49c2820c034f72b P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.911070 8418 raft_consensus.cc:374] T d1569c32127d4920b49c2820c034f72b P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:21.911105 8418 raft_consensus.cc:729] T d1569c32127d4920b49c2820c034f72b P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 019b6a6a8c374064be3f9293e1b62077, State: Initialized, Role: FOLLOWER
I20260926 11:30:21.911240 8418 consensus_queue.cc:260] T d1569c32127d4920b49c2820c034f72b P 019b6a6a8c374064be3f9293e1b62077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:21.911557 8418 ts_tablet_manager.cc:1428] T d1569c32127d4920b49c2820c034f72b P 019b6a6a8c374064be3f9293e1b62077: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 11:30:22.243681 8428 raft_consensus.cc:482] T d1569c32127d4920b49c2820c034f72b P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:30:22.243883 8428 raft_consensus.cc:504] T d1569c32127d4920b49c2820c034f72b P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:22.244359 8428 leader_election.cc:288] T d1569c32127d4920b49c2820c034f72b P 019b6a6a8c374064be3f9293e1b62077 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 604a0651b0d94c288f6dd4f7548f3c29 (127.0.98.2:36991), a4a63e8498964e47b64a3d761292bd49 (127.0.98.3:32989)
I20260926 11:30:22.244827 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1569c32127d4920b49c2820c034f72b" candidate_uuid: "019b6a6a8c374064be3f9293e1b62077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4a63e8498964e47b64a3d761292bd49" is_pre_election: true
I20260926 11:30:22.244851 8203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1569c32127d4920b49c2820c034f72b" candidate_uuid: "019b6a6a8c374064be3f9293e1b62077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "604a0651b0d94c288f6dd4f7548f3c29" is_pre_election: true
I20260926 11:30:22.244973 8343 raft_consensus.cc:2454] T d1569c32127d4920b49c2820c034f72b P a4a63e8498964e47b64a3d761292bd49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 019b6a6a8c374064be3f9293e1b62077 in term 0.
I20260926 11:30:22.245036 8203 raft_consensus.cc:2454] T d1569c32127d4920b49c2820c034f72b P 604a0651b0d94c288f6dd4f7548f3c29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 019b6a6a8c374064be3f9293e1b62077 in term 0.
I20260926 11:30:22.245319 7997 leader_election.cc:302] T d1569c32127d4920b49c2820c034f72b P 019b6a6a8c374064be3f9293e1b62077 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 019b6a6a8c374064be3f9293e1b62077, a4a63e8498964e47b64a3d761292bd49; no voters:
I20260926 11:30:22.245484 8428 raft_consensus.cc:2781] T d1569c32127d4920b49c2820c034f72b P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:30:22.245543 8428 raft_consensus.cc:482] T d1569c32127d4920b49c2820c034f72b P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:30:22.245569 8428 raft_consensus.cc:3037] T d1569c32127d4920b49c2820c034f72b P 019b6a6a8c374064be3f9293e1b62077 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:22.246722 8428 raft_consensus.cc:504] T d1569c32127d4920b49c2820c034f72b P 019b6a6a8c374064be3f9293e1b62077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:22.247082 8428 leader_election.cc:288] T d1569c32127d4920b49c2820c034f72b P 019b6a6a8c374064be3f9293e1b62077 [CANDIDATE]: Term 1 election: Requested vote from peers 604a0651b0d94c288f6dd4f7548f3c29 (127.0.98.2:36991), a4a63e8498964e47b64a3d761292bd49 (127.0.98.3:32989)
I20260926 11:30:22.247435 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1569c32127d4920b49c2820c034f72b" candidate_uuid: "019b6a6a8c374064be3f9293e1b62077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4a63e8498964e47b64a3d761292bd49"
I20260926 11:30:22.247433 8203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1569c32127d4920b49c2820c034f72b" candidate_uuid: "019b6a6a8c374064be3f9293e1b62077" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "604a0651b0d94c288f6dd4f7548f3c29"
I20260926 11:30:22.247552 8343 raft_consensus.cc:3037] T d1569c32127d4920b49c2820c034f72b P a4a63e8498964e47b64a3d761292bd49 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:22.247560 8203 raft_consensus.cc:3037] T d1569c32127d4920b49c2820c034f72b P 604a0651b0d94c288f6dd4f7548f3c29 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:22.248468 8203 raft_consensus.cc:2454] T d1569c32127d4920b49c2820c034f72b P 604a0651b0d94c288f6dd4f7548f3c29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 019b6a6a8c374064be3f9293e1b62077 in term 1.
I20260926 11:30:22.248478 8343 raft_consensus.cc:2454] T d1569c32127d4920b49c2820c034f72b P a4a63e8498964e47b64a3d761292bd49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 019b6a6a8c374064be3f9293e1b62077 in term 1.
I20260926 11:30:22.248754 7997 leader_election.cc:302] T d1569c32127d4920b49c2820c034f72b P 019b6a6a8c374064be3f9293e1b62077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 019b6a6a8c374064be3f9293e1b62077, a4a63e8498964e47b64a3d761292bd49; no voters:
I20260926 11:30:22.248937 8428 raft_consensus.cc:2781] T d1569c32127d4920b49c2820c034f72b P 019b6a6a8c374064be3f9293e1b62077 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:30:22.249101 8428 raft_consensus.cc:686] T d1569c32127d4920b49c2820c034f72b P 019b6a6a8c374064be3f9293e1b62077 [term 1 LEADER]: Becoming Leader. State: Replica: 019b6a6a8c374064be3f9293e1b62077, State: Running, Role: LEADER
I20260926 11:30:22.249248 8428 consensus_queue.cc:237] T d1569c32127d4920b49c2820c034f72b P 019b6a6a8c374064be3f9293e1b62077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } }
I20260926 11:30:22.250767 7905 catalog_manager.cc:5495] T d1569c32127d4920b49c2820c034f72b P 019b6a6a8c374064be3f9293e1b62077 reported cstate change: term changed from 0 to 1, leader changed from <none> to 019b6a6a8c374064be3f9293e1b62077 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "019b6a6a8c374064be3f9293e1b62077" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "019b6a6a8c374064be3f9293e1b62077" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 } health_report { overall_health: UNKNOWN } } }
I20260926 11:30:22.283123 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:22.280237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60990 (local address 127.0.98.1:41563)
0926 11:30:22.280359 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:22.280363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:22.280377 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:30:22.280512 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:22.280514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:22.280553 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:22.280613 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:30:22.280801 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:22.281354 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:22.282148 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:22.282405 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:22.282529 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:22.282736 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:30:22.282831 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:30:22.282890 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:22.282978 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 11:30:22.283021 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 11:30:22.288663 8203 raft_consensus.cc:1261] T d1569c32127d4920b49c2820c034f72b P 604a0651b0d94c288f6dd4f7548f3c29 [term 1 FOLLOWER]: Refusing update from remote peer 019b6a6a8c374064be3f9293e1b62077: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:30:22.288666 8343 raft_consensus.cc:1261] T d1569c32127d4920b49c2820c034f72b P a4a63e8498964e47b64a3d761292bd49 [term 1 FOLLOWER]: Refusing update from remote peer 019b6a6a8c374064be3f9293e1b62077: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:30:22.289043 8500 consensus_queue.cc:1035] T d1569c32127d4920b49c2820c034f72b P 019b6a6a8c374064be3f9293e1b62077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "604a0651b0d94c288f6dd4f7548f3c29" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:30:22.289194 8498 consensus_queue.cc:1035] T d1569c32127d4920b49c2820c034f72b P 019b6a6a8c374064be3f9293e1b62077 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4a63e8498964e47b64a3d761292bd49" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:30:22.320590 7905 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:30:22.320739 7905 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:30:22.322611 7905 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c53348dae04456b9c59629c03d412d6: Sending DeleteTablet for 3 replicas of tablet d1569c32127d4920b49c2820c034f72b
I20260926 11:30:22.323156 8042 tablet_service.cc:1514] Processing DeleteTablet for tablet d1569c32127d4920b49c2820c034f72b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:30:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60964
I20260926 11:30:22.323257 8323 tablet_service.cc:1514] Processing DeleteTablet for tablet d1569c32127d4920b49c2820c034f72b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:30:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46356
I20260926 11:30:22.323323 8165 tablet_service.cc:1514] Processing DeleteTablet for tablet d1569c32127d4920b49c2820c034f72b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:30:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35382
I20260926 11:30:22.323436 8442 tablet_replica.cc:330] T d1569c32127d4920b49c2820c034f72b P a4a63e8498964e47b64a3d761292bd49: stopping tablet replica
I20260926 11:30:22.323457 8443 tablet_replica.cc:330] T d1569c32127d4920b49c2820c034f72b P 019b6a6a8c374064be3f9293e1b62077: stopping tablet replica
I20260926 11:30:22.323536 8441 tablet_replica.cc:330] T d1569c32127d4920b49c2820c034f72b P 604a0651b0d94c288f6dd4f7548f3c29: stopping tablet replica
I20260926 11:30:22.323560 8442 raft_consensus.cc:2229] T d1569c32127d4920b49c2820c034f72b P a4a63e8498964e47b64a3d761292bd49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:30:22.323560 8443 raft_consensus.cc:2229] T d1569c32127d4920b49c2820c034f72b P 019b6a6a8c374064be3f9293e1b62077 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:30:22.323678 8442 raft_consensus.cc:2258] T d1569c32127d4920b49c2820c034f72b P a4a63e8498964e47b64a3d761292bd49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:30:22.323664 8441 raft_consensus.cc:2229] T d1569c32127d4920b49c2820c034f72b P 604a0651b0d94c288f6dd4f7548f3c29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:30:22.323801 8443 raft_consensus.cc:2258] T d1569c32127d4920b49c2820c034f72b P 019b6a6a8c374064be3f9293e1b62077 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:30:22.323819 8441 raft_consensus.cc:2258] T d1569c32127d4920b49c2820c034f72b P 604a0651b0d94c288f6dd4f7548f3c29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:30:22.324622 8443 ts_tablet_manager.cc:1905] T d1569c32127d4920b49c2820c034f72b P 019b6a6a8c374064be3f9293e1b62077: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:30:22.324622 8442 ts_tablet_manager.cc:1905] T d1569c32127d4920b49c2820c034f72b P a4a63e8498964e47b64a3d761292bd49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:30:22.324796 8441 ts_tablet_manager.cc:1905] T d1569c32127d4920b49c2820c034f72b P 604a0651b0d94c288f6dd4f7548f3c29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:30:22.326911 8441 ts_tablet_manager.cc:1918] T d1569c32127d4920b49c2820c034f72b P 604a0651b0d94c288f6dd4f7548f3c29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 11:30:22.327005 8441 log.cc:1196] T d1569c32127d4920b49c2820c034f72b P 604a0651b0d94c288f6dd4f7548f3c29: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-1/wal/wals/d1569c32127d4920b49c2820c034f72b
I20260926 11:30:22.327086 8442 ts_tablet_manager.cc:1918] T d1569c32127d4920b49c2820c034f72b P a4a63e8498964e47b64a3d761292bd49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 11:30:22.327086 8443 ts_tablet_manager.cc:1918] T d1569c32127d4920b49c2820c034f72b P 019b6a6a8c374064be3f9293e1b62077: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 11:30:22.327170 8442 log.cc:1196] T d1569c32127d4920b49c2820c034f72b P a4a63e8498964e47b64a3d761292bd49: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-2/wal/wals/d1569c32127d4920b49c2820c034f72b
I20260926 11:30:22.327171 8443 log.cc:1196] T d1569c32127d4920b49c2820c034f72b P 019b6a6a8c374064be3f9293e1b62077: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790422180210734-392-0/minicluster-data/ts-0/wal/wals/d1569c32127d4920b49c2820c034f72b
I20260926 11:30:22.327292 8441 ts_tablet_manager.cc:1939] T d1569c32127d4920b49c2820c034f72b P 604a0651b0d94c288f6dd4f7548f3c29: Deleting consensus metadata
I20260926 11:30:22.327441 8442 ts_tablet_manager.cc:1939] T d1569c32127d4920b49c2820c034f72b P a4a63e8498964e47b64a3d761292bd49: Deleting consensus metadata
I20260926 11:30:22.327441 8443 ts_tablet_manager.cc:1939] T d1569c32127d4920b49c2820c034f72b P 019b6a6a8c374064be3f9293e1b62077: Deleting consensus metadata
I20260926 11:30:22.328176 7891 catalog_manager.cc:4841] TS 019b6a6a8c374064be3f9293e1b62077 (127.0.98.1:41563): tablet d1569c32127d4920b49c2820c034f72b (table test-table [id=d2528f88a93442ea977d8635f0cca855]) successfully deleted
I20260926 11:30:22.328368 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 7978
I20260926 11:30:22.328408 7893 catalog_manager.cc:4841] TS a4a63e8498964e47b64a3d761292bd49 (127.0.98.3:32989): tablet d1569c32127d4920b49c2820c034f72b (table test-table [id=d2528f88a93442ea977d8635f0cca855]) successfully deleted
I20260926 11:30:22.332063 7890 catalog_manager.cc:4841] TS 604a0651b0d94c288f6dd4f7548f3c29 (127.0.98.2:36991): tablet d1569c32127d4920b49c2820c034f72b (table test-table [id=d2528f88a93442ea977d8635f0cca855]) successfully deleted
I20260926 11:30:22.336715 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 8118
I20260926 11:30:22.342655 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 8258
I20260926 11:30:22.349184 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 7874
2026-09-26T11:30:22Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3514 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:30:22 dist-test-slave-mrck krb5kdc[8507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:30:22 dist-test-slave-mrck krb5kdc[8507](info): set up 2 sockets
Sep 26 11:30:22 dist-test-slave-mrck krb5kdc[8507](info): commencing operation
krb5kdc: starting...
W20260926 11:30:23.384483 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:30:23 dist-test-slave-mrck krb5kdc[8507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:30:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:30:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:30:23.503252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35047
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35047
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:23.613785 8520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:23.614089 8520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:23.614130 8520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:23.618278 8520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:30:23.618402 8520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:23.618435 8520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:23.618453 8520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:30:23.618468 8520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:30:23.624082 8520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:23.625211 8520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:23.626225 8520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:23.634955 8528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:23.634940 8525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:23.634979 8520 server_base.cc:1034] running on GCE node
W20260926 11:30:23.634938 8526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:23.635850 8520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:23.636996 8520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:23.638200 8520 hybrid_clock.cc:648] HybridClock initialized: now 1790422223638178 us; error 44 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T11:30:23Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1301 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:30:23 dist-test-slave-mrck krb5kdc[8533](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:30:23 dist-test-slave-mrck krb5kdc[8533](info): set up 2 sockets
Sep 26 11:30:23 dist-test-slave-mrck krb5kdc[8533](info): commencing operation
krb5kdc: starting...
W20260926 11:30:24.684490 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:30:24 dist-test-slave-mrck krb5kdc[8533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:30:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:30:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:30:24.810776 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36565
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:24.920143 8546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:24.920419 8546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:24.920459 8546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:24.924521 8546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:30:24.924589 8546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:24.924609 8546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:24.924626 8546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:30:24.924641 8546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:30:24.930095 8546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:24.931219 8546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:24.932348 8546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 11:30:24.940665 8546 server_base.cc:1034] running on GCE node
W20260926 11:30:24.940626 8554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:24.940632 8551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:24.940630 8552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:24.941362 8546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:24.942457 8546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:24.943630 8546 hybrid_clock.cc:648] HybridClock initialized: now 1790422224943605 us; error 44 us; skew 500 ppm
Sep 26 11:30:24 dist-test-slave-mrck krb5kdc[8533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:24.946324 8546 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:30:24.947239 8546 webserver.cc:460] Webserver started at http://127.0.98.62:40891/ using document root <none> and password file <none>
I20260926 11:30:24.947877 8546 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:24.947965 8546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:24.948174 8546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:24.950217 8546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "58c8ce9bc86141eb99124092eef4b5ba"
format_stamp: "Formatted at 2026-09-26 11:30:24 on dist-test-slave-mrck"
server_key: "347c6f6b7032b497ee8f668bede40b90"
server_key_iv: "5f48a1a4faf3981b9e6d208b7bd6e070"
server_key_version: "encryptionkey@0"
I20260926 11:30:24.950775 8546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "58c8ce9bc86141eb99124092eef4b5ba"
format_stamp: "Formatted at 2026-09-26 11:30:24 on dist-test-slave-mrck"
server_key: "347c6f6b7032b497ee8f668bede40b90"
server_key_iv: "5f48a1a4faf3981b9e6d208b7bd6e070"
server_key_version: "encryptionkey@0"
I20260926 11:30:24.954493 8546 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 11:30:24.956990 8561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:24.958156 8546 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 11:30:24.958289 8546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "58c8ce9bc86141eb99124092eef4b5ba"
format_stamp: "Formatted at 2026-09-26 11:30:24 on dist-test-slave-mrck"
server_key: "347c6f6b7032b497ee8f668bede40b90"
server_key_iv: "5f48a1a4faf3981b9e6d208b7bd6e070"
server_key_version: "encryptionkey@0"
I20260926 11:30:24.958420 8546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:24.968453 8546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:24.971218 8546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:24.971411 8546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:24.978163 8546 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36565
I20260926 11:30:24.978184 8613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36565 every 8 connection(s)
I20260926 11:30:24.979131 8546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:30:24.981803 8614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:24.987113 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 8546
I20260926 11:30:24.987138 8614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba: Bootstrap starting.
I20260926 11:30:24.987221 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:30:24.987496 392 external_mini_cluster.cc:1428] Setting key 1e5645415a189ebdc4a54ca1c7ce21ba
I20260926 11:30:24.989904 8614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:24.990775 8614 log.cc:824] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba: Log is configured to *not* fsync() on all Append() calls
I20260926 11:30:24.993007 8614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba: No bootstrap required, opened a new log
Sep 26 11:30:24 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:30:24.997184 8614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c8ce9bc86141eb99124092eef4b5ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36565 } }
I20260926 11:30:24.997483 8614 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:24.997547 8614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58c8ce9bc86141eb99124092eef4b5ba, State: Initialized, Role: FOLLOWER
I20260926 11:30:24.998034 8614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c8ce9bc86141eb99124092eef4b5ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36565 } }
I20260926 11:30:24.998185 8614 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:30:24.998266 8614 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:30:24.998373 8614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:24.999428 8614 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c8ce9bc86141eb99124092eef4b5ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36565 } }
I20260926 11:30:24.999828 8614 leader_election.cc:302] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58c8ce9bc86141eb99124092eef4b5ba; no voters:
I20260926 11:30:25.000212 8614 leader_election.cc:288] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:30:25.000313 8619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:30:25.000531 8619 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [term 1 LEADER]: Becoming Leader. State: Replica: 58c8ce9bc86141eb99124092eef4b5ba, State: Running, Role: LEADER
I20260926 11:30:25.000850 8619 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c8ce9bc86141eb99124092eef4b5ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36565 } }
I20260926 11:30:25.001631 8614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:30:25.003551 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:24.988920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42220 (local address 127.0.98.62:36565)
0926 11:30:24.989384 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:24.989403 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:24.989446 (+ 43us) server_negotiation.cc:407] Connection header received
0926 11:30:24.990325 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:24.990349 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:24.990776 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:24.991149 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:24.992005 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:24.993465 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:24.994280 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:24.994999 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:24.995159 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:24.997864 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:24.997870 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:24.997887 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:24.997922 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:24.999701 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:25.000253 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:25.000259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:25.000266 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:25.000418 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:25.000790 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:25.000793 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:25.000795 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:25.001294 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:25.001487 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:25.002126 (+ 639us) server_negotiation.cc:299] Negotiation successful
0926 11:30:25.002466 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":135,"threads_started":1}
I20260926 11:30:25.004262 8621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58c8ce9bc86141eb99124092eef4b5ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c8ce9bc86141eb99124092eef4b5ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36565 } } }
I20260926 11:30:25.004402 8621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:25.004244 8620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58c8ce9bc86141eb99124092eef4b5ba. Latest consensus state: current_term: 1 leader_uuid: "58c8ce9bc86141eb99124092eef4b5ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c8ce9bc86141eb99124092eef4b5ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36565 } } }
I20260926 11:30:25.004652 8620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:25.005594 8634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:30:25.008183 8634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:30:25.013175 8634 catalog_manager.cc:1347] Generated new cluster ID: 82d2870e4df340a5b0c1b162675095aa
I20260926 11:30:25.013242 8634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:30:25.051400 8634 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:30:25.052799 8634 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:30:25.060379 8634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba: Generated new TSK 0
I20260926 11:30:25.061097 8634 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:30:25.098459 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36565
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:25.210539 8640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:25.210811 8640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:25.210857 8640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:25.215008 8640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:25.215070 8640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:25.215128 8640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:30:25.220397 8640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:25.221360 8640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:25.222299 8640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 11:30:25.232059 8640 server_base.cc:1034] running on GCE node
W20260926 11:30:25.232081 8645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:25.232092 8648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:25.232303 8646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:25.232834 8640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:25.233418 8640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:25.234591 8640 hybrid_clock.cc:648] HybridClock initialized: now 1790422225234573 us; error 32 us; skew 500 ppm
Sep 26 11:30:25 dist-test-slave-mrck krb5kdc[8533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:25.237432 8640 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:30:25.238332 8640 webserver.cc:460] Webserver started at http://127.0.98.1:33175/ using document root <none> and password file <none>
I20260926 11:30:25.238879 8640 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:25.238961 8640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:25.239193 8640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:25.241410 8640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-0/data/instance:
uuid: "256d0d3f1e04495d8829117b288dfc1f"
format_stamp: "Formatted at 2026-09-26 11:30:25 on dist-test-slave-mrck"
server_key: "b6516dee020f39b67009a58b9e52f3c0"
server_key_iv: "26e54fa7f5a6fecc0577273f06a8e540"
server_key_version: "encryptionkey@0"
I20260926 11:30:25.241955 8640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "256d0d3f1e04495d8829117b288dfc1f"
format_stamp: "Formatted at 2026-09-26 11:30:25 on dist-test-slave-mrck"
server_key: "b6516dee020f39b67009a58b9e52f3c0"
server_key_iv: "26e54fa7f5a6fecc0577273f06a8e540"
server_key_version: "encryptionkey@0"
I20260926 11:30:25.245739 8640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 11:30:25.248260 8655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:25.249428 8640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:30:25.249552 8640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "256d0d3f1e04495d8829117b288dfc1f"
format_stamp: "Formatted at 2026-09-26 11:30:25 on dist-test-slave-mrck"
server_key: "b6516dee020f39b67009a58b9e52f3c0"
server_key_iv: "26e54fa7f5a6fecc0577273f06a8e540"
server_key_version: "encryptionkey@0"
I20260926 11:30:25.249665 8640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:25.259899 8640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:25.263034 8640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:25.263232 8640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:25.263862 8640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:25.264708 8640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:25.264770 8640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:25.264837 8640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:25.264880 8640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:25.273676 8640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46139
I20260926 11:30:25.273695 8768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46139 every 8 connection(s)
I20260926 11:30:25.274575 8640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 11:30:25 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:30:25.284825 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 8640
I20260926 11:30:25.284960 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:30:25.285255 392 external_mini_cluster.cc:1428] Setting key 9c7b47c42825139c5a238fa1b478d9ea
I20260926 11:30:25.288635 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:25.276351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60801 (local address 127.0.98.62:36565)
0926 11:30:25.276481 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:25.276484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:25.277464 (+ 980us) server_negotiation.cc:407] Connection header received
0926 11:30:25.278883 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:25.278886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:25.278944 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:25.279040 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:25.280259 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:25.280790 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:25.281914 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:25.282145 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:25.282254 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:25.284789 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:25.284792 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:25.284794 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:25.284818 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:25.286475 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:25.287026 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:25.287030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:25.287032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:25.287125 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:25.287434 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:25.287440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:25.287445 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:25.287709 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:25.287836 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:25.288349 (+ 513us) server_negotiation.cc:299] Negotiation successful
0926 11:30:25.288490 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 11:30:25.289868 8771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:25.276721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.1:60801)
0926 11:30:25.277304 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:25.277349 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:25.278661 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:25.279210 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:25.279218 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:25.279579 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:25.280119 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:25.280132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:25.280911 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:25.280915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:25.281796 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:25.281804 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:25.282305 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:25.282309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:25.282467 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:25.283056 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:25.283073 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:25.284619 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:25.286610 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:25.286615 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:25.286632 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:25.286930 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:25.287208 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:25.287210 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:25.287211 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:25.287349 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:25.287824 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:25.287829 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:25.288090 (+ 261us) client_negotiation.cc:769] Sending connection context
0926 11:30:25.288315 (+ 225us) client_negotiation.cc:240] Negotiation successful
0926 11:30:25.288728 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":108,"threads_started":1}
I20260926 11:30:25.291093 8769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36565
I20260926 11:30:25.291366 8769 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:25.291960 8769 heartbeater.cc:511] Master 127.0.98.62:36565 requested a full tablet report, sending...
I20260926 11:30:25.293466 8578 ts_manager.cc:194] Registered new tserver with Master: 256d0d3f1e04495d8829117b288dfc1f (127.0.98.1:46139)
I20260926 11:30:25.294596 8578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60801
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:30:25.312341 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36565
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:25.423691 8774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:25.423985 8774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:25.424065 8774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:25.428407 8774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:25.428463 8774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:25.428560 8774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:30:25.433900 8774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:25.434998 8774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:25.436214 8774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:25.446133 8780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:25.446142 8782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:25.446225 8779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:25.446277 8774 server_base.cc:1034] running on GCE node
I20260926 11:30:25.446853 8774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:25.447438 8774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:25.448589 8774 hybrid_clock.cc:648] HybridClock initialized: now 1790422225448540 us; error 62 us; skew 500 ppm
Sep 26 11:30:25 dist-test-slave-mrck krb5kdc[8533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:25.451205 8774 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:30:25.452071 8774 webserver.cc:460] Webserver started at http://127.0.98.2:36361/ using document root <none> and password file <none>
I20260926 11:30:25.452582 8774 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:25.452637 8774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:25.452818 8774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:25.454846 8774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-1/data/instance:
uuid: "7e2941485a7944469d41a2af0c3bfa92"
format_stamp: "Formatted at 2026-09-26 11:30:25 on dist-test-slave-mrck"
server_key: "1b4da1d6fab54b86ee9f06316b63436c"
server_key_iv: "1d33d046ae4ae0fa1fd25a8eca6d9673"
server_key_version: "encryptionkey@0"
I20260926 11:30:25.455356 8774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7e2941485a7944469d41a2af0c3bfa92"
format_stamp: "Formatted at 2026-09-26 11:30:25 on dist-test-slave-mrck"
server_key: "1b4da1d6fab54b86ee9f06316b63436c"
server_key_iv: "1d33d046ae4ae0fa1fd25a8eca6d9673"
server_key_version: "encryptionkey@0"
I20260926 11:30:25.458917 8774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 11:30:25.461236 8789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:25.462489 8774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 11:30:25.462575 8774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "7e2941485a7944469d41a2af0c3bfa92"
format_stamp: "Formatted at 2026-09-26 11:30:25 on dist-test-slave-mrck"
server_key: "1b4da1d6fab54b86ee9f06316b63436c"
server_key_iv: "1d33d046ae4ae0fa1fd25a8eca6d9673"
server_key_version: "encryptionkey@0"
I20260926 11:30:25.462654 8774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:25.476631 8774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:25.479476 8774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:25.479624 8774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:25.480214 8774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:25.480979 8774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:25.481025 8774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:25.481063 8774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:25.481086 8774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:25.489604 8774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41051
I20260926 11:30:25.489624 8902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41051 every 8 connection(s)
I20260926 11:30:25.490549 8774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:30:25.498522 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 8774
I20260926 11:30:25.498664 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:30:25.498970 392 external_mini_cluster.cc:1428] Setting key 31678bfcd09f61acc4b52c1b41496946
Sep 26 11:30:25 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:30:25.504490 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:25.492309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35085 (local address 127.0.98.62:36565)
0926 11:30:25.492441 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:25.492445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:25.493294 (+ 849us) server_negotiation.cc:407] Connection header received
0926 11:30:25.494577 (+ 1283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:25.494580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:25.494624 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:25.494699 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:25.495849 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:25.496364 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:25.497553 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:25.497764 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:25.497833 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:25.500545 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:25.500548 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:25.500550 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:25.500571 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:25.502158 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:25.502778 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:25.502784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:25.502789 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:25.502886 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:25.503270 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:25.503274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:25.503276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:25.503486 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:25.503579 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:25.504193 (+ 614us) server_negotiation.cc:299] Negotiation successful
0926 11:30:25.504323 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 11:30:25.505393 8905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:25.492707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.2:35085)
0926 11:30:25.493130 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:25.493171 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:25.494396 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:25.494852 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:25.494859 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:25.495190 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:25.495677 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:25.495689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:25.496506 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:25.496509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:25.497441 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:25.497451 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:25.497889 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:25.497892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:25.498036 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:25.498751 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:25.498776 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:25.500375 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:25.502278 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:25.502285 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:25.502304 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:25.502665 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:25.502989 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:25.502995 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:25.503016 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:25.503168 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:25.503576 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:25.503583 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:25.503919 (+ 336us) client_negotiation.cc:769] Sending connection context
0926 11:30:25.504143 (+ 224us) client_negotiation.cc:240] Negotiation successful
0926 11:30:25.504464 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":109,"threads_started":1}
I20260926 11:30:25.506557 8903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36565
I20260926 11:30:25.506844 8903 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:25.507450 8903 heartbeater.cc:511] Master 127.0.98.62:36565 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 11:30:25.508589 8578 ts_manager.cc:194] Registered new tserver with Master: 7e2941485a7944469d41a2af0c3bfa92 (127.0.98.2:41051)
I20260926 11:30:25.509133 8578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:30:25.525019 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36565
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:25.635308 8908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:25.635573 8908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:25.635617 8908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:25.639698 8908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:25.639839 8908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:25.639904 8908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:30:25.645339 8908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:25.646450 8908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:25.647445 8908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:25.656935 8914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:25.656920 8916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:25.656920 8913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:25.657248 8908 server_base.cc:1034] running on GCE node
I20260926 11:30:25.657707 8908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:25.658272 8908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:25.659451 8908 hybrid_clock.cc:648] HybridClock initialized: now 1790422225659434 us; error 33 us; skew 500 ppm
Sep 26 11:30:25 dist-test-slave-mrck krb5kdc[8533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:25.662143 8908 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:30:25.663081 8908 webserver.cc:460] Webserver started at http://127.0.98.3:42299/ using document root <none> and password file <none>
I20260926 11:30:25.663623 8908 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:25.663708 8908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:25.663955 8908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:25.665905 8908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-2/data/instance:
uuid: "fb4a9e1560f84bf1a6eb1ac480f6e6f0"
format_stamp: "Formatted at 2026-09-26 11:30:25 on dist-test-slave-mrck"
server_key: "6d1ea0680aa15ea53d099e5d8f3e3be9"
server_key_iv: "9fd0186258c9fca473b7de98cad86397"
server_key_version: "encryptionkey@0"
I20260926 11:30:25.666495 8908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fb4a9e1560f84bf1a6eb1ac480f6e6f0"
format_stamp: "Formatted at 2026-09-26 11:30:25 on dist-test-slave-mrck"
server_key: "6d1ea0680aa15ea53d099e5d8f3e3be9"
server_key_iv: "9fd0186258c9fca473b7de98cad86397"
server_key_version: "encryptionkey@0"
I20260926 11:30:25.670023 8908 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 11:30:25.672423 8923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:25.673591 8908 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:30:25.673710 8908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "fb4a9e1560f84bf1a6eb1ac480f6e6f0"
format_stamp: "Formatted at 2026-09-26 11:30:25 on dist-test-slave-mrck"
server_key: "6d1ea0680aa15ea53d099e5d8f3e3be9"
server_key_iv: "9fd0186258c9fca473b7de98cad86397"
server_key_version: "encryptionkey@0"
I20260926 11:30:25.673807 8908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:25.706836 8908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:25.709829 8908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:25.710027 8908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:25.710613 8908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:25.711488 8908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:25.711561 8908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:25.711628 8908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:25.711674 8908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:25.720582 8908 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37875
I20260926 11:30:25.720599 9036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37875 every 8 connection(s)
I20260926 11:30:25.721529 8908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 11:30:25 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:30:25.731619 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 8908
I20260926 11:30:25.731796 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:30:25.732215 392 external_mini_cluster.cc:1428] Setting key 47348a42208b748f1723b477a51411c3
I20260926 11:30:25.735805 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:25.723506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48655 (local address 127.0.98.62:36565)
0926 11:30:25.723679 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:25.723684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:25.724648 (+ 964us) server_negotiation.cc:407] Connection header received
0926 11:30:25.726044 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:25.726048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:25.726106 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:25.726208 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:25.727353 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:25.727952 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:25.729084 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:25.729316 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:25.729421 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:25.732112 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:25.732115 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:25.732117 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:25.732141 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:25.734010 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:25.734520 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:25.734523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:25.734524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:25.734599 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:25.734866 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:25.734868 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:25.734869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:25.735018 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:25.735084 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:25.735547 (+ 463us) server_negotiation.cc:299] Negotiation successful
0926 11:30:25.735641 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 11:30:25.736666 9039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:25.724033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.3:48655)
0926 11:30:25.724482 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:25.724524 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:25.725833 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:25.726385 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:25.726392 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:25.726741 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:25.727211 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:25.727224 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:25.728068 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:25.728072 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:25.728959 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:25.728968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:25.729458 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:25.729462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:25.729632 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:25.730210 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:25.730225 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:25.731938 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:25.734113 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:25.734118 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:25.734134 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:25.734439 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:25.734673 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:25.734675 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:25.734676 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:25.734810 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:25.735089 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:25.735094 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:25.735353 (+ 259us) client_negotiation.cc:769] Sending connection context
0926 11:30:25.735506 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 11:30:25.735827 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":116,"threads_started":1}
I20260926 11:30:25.737661 9037 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36565
I20260926 11:30:25.737860 9037 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:25.738385 9037 heartbeater.cc:511] Master 127.0.98.62:36565 requested a full tablet report, sending...
I20260926 11:30:25.739259 8578 ts_manager.cc:194] Registered new tserver with Master: fb4a9e1560f84bf1a6eb1ac480f6e6f0 (127.0.98.3:37875)
I20260926 11:30:25.739866 8578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48655
I20260926 11:30:25.745995 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 11:30:25.813830 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 8640
I20260926 11:30:25.819432 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 8774
I20260926 11:30:25.824923 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 8908
I20260926 11:30:25.830204 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 8546
I20260926 11:30:25.836206 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36565
--webserver_interface=127.0.98.62
--webserver_port=40891
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36565
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:25.945467 9046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:25.945809 9046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:25.945912 9046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:25.949967 9046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:30:25.950062 9046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:25.950100 9046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:25.950136 9046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:30:25.950173 9046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:30:25.955451 9046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:25.956573 9046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:25.957602 9046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:25.965804 9054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:25.965857 9051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:25.965911 9046 server_base.cc:1034] running on GCE node
W20260926 11:30:25.965849 9052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:25.966606 9046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:25.967679 9046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:25.968874 9046 hybrid_clock.cc:648] HybridClock initialized: now 1790422225968858 us; error 30 us; skew 500 ppm
Sep 26 11:30:25 dist-test-slave-mrck krb5kdc[8533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:25.971416 9046 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:30:25.972311 9046 webserver.cc:460] Webserver started at http://127.0.98.62:40891/ using document root <none> and password file <none>
I20260926 11:30:25.972856 9046 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:25.972956 9046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:25.976740 9046 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:30:25.978842 9061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:25.980165 9046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:30:25.980319 9046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "58c8ce9bc86141eb99124092eef4b5ba"
format_stamp: "Formatted at 2026-09-26 11:30:24 on dist-test-slave-mrck"
server_key: "347c6f6b7032b497ee8f668bede40b90"
server_key_iv: "5f48a1a4faf3981b9e6d208b7bd6e070"
server_key_version: "encryptionkey@0"
I20260926 11:30:25.980847 9046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:25.994023 9046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:25.997129 9046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:25.997357 9046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:26.004662 9046 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36565
I20260926 11:30:26.004683 9113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36565 every 8 connection(s)
I20260926 11:30:26.005800 9046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:30:26.010020 9114 sys_catalog.cc:263] Verifying existing consensus state
I20260926 11:30:26.011606 9114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba: Bootstrap starting.
I20260926 11:30:26.012089 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 9046
I20260926 11:30:26.012570 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46139
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36565
--webserver_port=33175
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:30:26.028898 9114 log.cc:824] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba: Log is configured to *not* fsync() on all Append() calls
I20260926 11:30:26.035419 9114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 11:30:26.035859 9114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba: Bootstrap complete.
I20260926 11:30:26.040100 9114 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c8ce9bc86141eb99124092eef4b5ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36565 } }
I20260926 11:30:26.040498 9114 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58c8ce9bc86141eb99124092eef4b5ba, State: Initialized, Role: FOLLOWER
I20260926 11:30:26.040989 9114 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c8ce9bc86141eb99124092eef4b5ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36565 } }
I20260926 11:30:26.041100 9114 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:30:26.041182 9114 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:30:26.041277 9114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [term 1 FOLLOWER]: Advancing to term 2
I20260926 11:30:26.043287 9114 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c8ce9bc86141eb99124092eef4b5ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36565 } }
I20260926 11:30:26.043612 9114 leader_election.cc:302] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58c8ce9bc86141eb99124092eef4b5ba; no voters:
I20260926 11:30:26.043963 9114 leader_election.cc:288] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 11:30:26.044077 9118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [term 2 FOLLOWER]: Leader election won for term 2
I20260926 11:30:26.044291 9118 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [term 2 LEADER]: Becoming Leader. State: Replica: 58c8ce9bc86141eb99124092eef4b5ba, State: Running, Role: LEADER
I20260926 11:30:26.044637 9118 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c8ce9bc86141eb99124092eef4b5ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36565 } }
I20260926 11:30:26.045202 9114 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:30:26.045583 9118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58c8ce9bc86141eb99124092eef4b5ba. Latest consensus state: current_term: 2 leader_uuid: "58c8ce9bc86141eb99124092eef4b5ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c8ce9bc86141eb99124092eef4b5ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36565 } } }
I20260926 11:30:26.045775 9118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:26.045967 9119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "58c8ce9bc86141eb99124092eef4b5ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58c8ce9bc86141eb99124092eef4b5ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36565 } } }
I20260926 11:30:26.046142 9119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:26.046252 9122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:30:26.049615 9122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:30:26.050354 9122 catalog_manager.cc:1259] Loaded cluster ID: 82d2870e4df340a5b0c1b162675095aa
I20260926 11:30:26.050416 9122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:30:26.052706 9122 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:30:26.053347 9122 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 58c8ce9bc86141eb99124092eef4b5ba: Loaded TSK: 0
I20260926 11:30:26.054540 9122 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 11:30:26.127215 9116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:26.127501 9116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:26.127575 9116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:26.131598 9116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:26.131692 9116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:26.131808 9116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:30:26.137187 9116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33175
--tserver_master_addrs=127.0.98.62:36565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:26.138669 9116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:26.140151 9116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:26.150667 9143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:26.150683 9140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:26.150818 9141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:26.150995 9116 server_base.cc:1034] running on GCE node
I20260926 11:30:26.151625 9116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:26.152439 9116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:26.153647 9116 hybrid_clock.cc:648] HybridClock initialized: now 1790422226153609 us; error 49 us; skew 500 ppm
Sep 26 11:30:26 dist-test-slave-mrck krb5kdc[8533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422226, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:26.157196 9116 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 11:30:26.158318 9116 webserver.cc:460] Webserver started at http://127.0.98.1:33175/ using document root <none> and password file <none>
I20260926 11:30:26.159024 9116 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:26.159129 9116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:26.164215 9116 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:30:26.166792 9150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:26.168371 9116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:30:26.168495 9116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "256d0d3f1e04495d8829117b288dfc1f"
format_stamp: "Formatted at 2026-09-26 11:30:25 on dist-test-slave-mrck"
server_key: "b6516dee020f39b67009a58b9e52f3c0"
server_key_iv: "26e54fa7f5a6fecc0577273f06a8e540"
server_key_version: "encryptionkey@0"
I20260926 11:30:26.169345 9116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:26.191244 9116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:26.195052 9116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:26.195240 9116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:26.195935 9116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:26.196918 9116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:26.196972 9116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:26.197021 9116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:26.197052 9116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:26.207402 9116 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46139
I20260926 11:30:26.207423 9263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46139 every 8 connection(s)
I20260926 11:30:26.208487 9116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:30:26.209492 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 9116
I20260926 11:30:26.209952 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41051
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36565
--webserver_port=36361
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 11:30:26 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:26 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 11:30:26.228852 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:26.210796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57233 (local address 127.0.98.62:36565)
0926 11:30:26.211280 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:26.211300 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:26.216371 (+ 5071us) server_negotiation.cc:407] Connection header received
0926 11:30:26.218641 (+ 2270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:26.218654 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:26.218972 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:26.219271 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:26.220916 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.221897 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.223549 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.224077 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.224219 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:26.228151 (+ 3932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57233: BlockingRecv error: recv got EOF from 127.0.98.1:57233 (error 108)
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":114,"threads_started":1}
I20260926 11:30:26.228892 9268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:26.210896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.1:57233)
0926 11:30:26.216132 (+ 5236us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:26.216202 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:26.218175 (+ 1973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:26.219423 (+ 1248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:26.219435 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:26.220046 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:26.220721 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:26.220739 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.222028 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.222031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:26.223411 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:26.223421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.224241 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.224245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:26.224437 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:26.225075 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:26.225095 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:26.227944 (+ 2849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":4968,"thread_start_us":131,"threads_started":1}
W20260926 11:30:26.229759 9264 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36565 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36565: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 11:30:26 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:26 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:26.235157 9268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:26.230209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.1:50007)
0926 11:30:26.230330 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:26.230347 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:26.230427 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:26.230655 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:26.230658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:26.230816 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:26.230905 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:26.230909 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.231616 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.231621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:26.232233 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:26.232241 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.232666 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.232669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:26.232777 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:26.233269 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:26.233279 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:26.235033 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 11:30:26.235494 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:26.230297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50007 (local address 127.0.98.62:36565)
0926 11:30:26.230413 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:26.230417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:26.230430 (+ 13us) server_negotiation.cc:407] Connection header received
0926 11:30:26.230508 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:26.230511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:26.230560 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:26.230642 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:26.231012 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.231515 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.232348 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.232563 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.232666 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:26.235196 (+ 2530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50007: BlockingRecv error: recv got EOF from 127.0.98.1:50007 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 26 11:30:26 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:26 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:26.240913 9268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:26.235723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.1:42039)
0926 11:30:26.235931 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:26.235950 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:26.236033 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:26.236471 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:26.236474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:26.236630 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:26.236703 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:26.236708 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.237470 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.237477 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:26.237998 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:26.238005 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.238423 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.238426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:26.238503 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:26.239014 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:26.239026 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:26.240781 (+ 1755us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 11:30:26.241003 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:26.235849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42039 (local address 127.0.98.62:36565)
0926 11:30:26.236002 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:26.236006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:26.236019 (+ 13us) server_negotiation.cc:407] Connection header received
0926 11:30:26.236134 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:26.236139 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:26.236326 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:26.236457 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:26.236809 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.237348 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.238111 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.238322 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.238425 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:26.240889 (+ 2464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42039: BlockingRecv error: recv got EOF from 127.0.98.1:42039 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260926 11:30:26.241231 9264 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 11:30:26.325400 9267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:26.325677 9267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:26.325721 9267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:26.329875 9267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:26.329947 9267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:26.330039 9267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:30:26.335602 9267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41051
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36361
--tserver_master_addrs=127.0.98.62:36565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:26.336917 9267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:26.337842 9267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:26.347121 9273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:26.347206 9274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:26.347209 9276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:26.347316 9267 server_base.cc:1034] running on GCE node
I20260926 11:30:26.347798 9267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:26.348501 9267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:26.349696 9267 hybrid_clock.cc:648] HybridClock initialized: now 1790422226349661 us; error 46 us; skew 500 ppm
Sep 26 11:30:26 dist-test-slave-mrck krb5kdc[8533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422226, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:26.352449 9267 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 11:30:26.353308 9267 webserver.cc:460] Webserver started at http://127.0.98.2:36361/ using document root <none> and password file <none>
I20260926 11:30:26.353863 9267 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:26.353962 9267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:26.357976 9267 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:30:26.360188 9283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:26.361398 9267 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:30:26.361541 9267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "7e2941485a7944469d41a2af0c3bfa92"
format_stamp: "Formatted at 2026-09-26 11:30:25 on dist-test-slave-mrck"
server_key: "1b4da1d6fab54b86ee9f06316b63436c"
server_key_iv: "1d33d046ae4ae0fa1fd25a8eca6d9673"
server_key_version: "encryptionkey@0"
I20260926 11:30:26.362082 9267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:26.382545 9267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:26.385869 9267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:26.386076 9267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:26.386766 9267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:26.387651 9267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:26.387722 9267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:26.387825 9267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:26.387890 9267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:26.397600 9267 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41051
I20260926 11:30:26.397625 9396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41051 every 8 connection(s)
I20260926 11:30:26.398645 9267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:30:26.406749 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 9267
I20260926 11:30:26.407179 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37875
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36565
--webserver_port=42299
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 11:30:26 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:26 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 11:30:26.410084 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:26.400650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57389 (local address 127.0.98.62:36565)
0926 11:30:26.400814 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:26.400817 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:26.401675 (+ 858us) server_negotiation.cc:407] Connection header received
0926 11:30:26.402978 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:26.402981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:26.403035 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:26.403145 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:26.404411 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.404941 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.406031 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.406252 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.406331 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:26.409975 (+ 3644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57389: BlockingRecv error: recv got EOF from 127.0.98.2:57389 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 11:30:26.410640 9399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:26.401029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.2:57389)
0926 11:30:26.401517 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:26.401559 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:26.402768 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:26.403342 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:26.403349 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:26.403705 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:26.404268 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:26.404282 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.405065 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.405068 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:26.405926 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:26.405935 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.406382 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.406385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:26.406517 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:26.407133 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:26.407156 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:26.409848 (+ 2692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":123,"threads_started":1}
W20260926 11:30:26.411142 9397 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36565 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36565: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 11:30:26 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:26 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:26.417673 9399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:26.411597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.2:34211)
0926 11:30:26.411718 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:26.411760 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:26.411835 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:26.412220 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:26.412223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:26.412377 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:26.412473 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:26.412479 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.413627 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.413630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:26.414164 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:26.414172 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.414799 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.414801 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:26.414872 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:26.415415 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:26.415428 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:26.417566 (+ 2138us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 11:30:26.417910 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:26.411802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34211 (local address 127.0.98.62:36565)
0926 11:30:26.411940 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:26.411947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:26.411964 (+ 17us) server_negotiation.cc:407] Connection header received
0926 11:30:26.412012 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:26.412017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:26.412073 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:26.412173 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:26.412589 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.413489 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.414275 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.414692 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.414817 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:26.417755 (+ 2938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34211: BlockingRecv error: recv got EOF from 127.0.98.2:34211 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 11:30:26 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:26 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:26.423504 9399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:26.418412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.2:57649)
0926 11:30:26.418533 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:26.418550 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:26.418638 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:26.418905 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:26.418908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:26.419063 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:26.419155 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:26.419161 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.419965 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.419974 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:26.420495 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:26.420503 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.420992 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.420996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:26.421083 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:26.421585 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:26.421597 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:26.423406 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 11:30:26.423722 9397 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 11:30:26.423803 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:26.418500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57649 (local address 127.0.98.62:36565)
0926 11:30:26.418638 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:26.418643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:26.418660 (+ 17us) server_negotiation.cc:407] Connection header received
0926 11:30:26.418752 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:26.418756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:26.418802 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:26.418891 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:26.419275 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.419848 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.420631 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.420852 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.420994 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:26.423549 (+ 2555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57649: BlockingRecv error: recv got EOF from 127.0.98.2:57649 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 11:30:26.520452 9400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:26.520731 9400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:26.520777 9400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:26.524863 9400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:26.524920 9400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:26.524981 9400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:30:26.530293 9400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42299
--tserver_master_addrs=127.0.98.62:36565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:26.531322 9400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:26.532334 9400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:26.541844 9408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:26.541844 9405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:26.542025 9400 server_base.cc:1034] running on GCE node
W20260926 11:30:26.541880 9406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:26.542520 9400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:26.543152 9400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:26.544404 9400 hybrid_clock.cc:648] HybridClock initialized: now 1790422226544368 us; error 53 us; skew 500 ppm
Sep 26 11:30:26 dist-test-slave-mrck krb5kdc[8533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422226, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:26.547358 9400 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 11:30:26.548321 9400 webserver.cc:460] Webserver started at http://127.0.98.3:42299/ using document root <none> and password file <none>
I20260926 11:30:26.548861 9400 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:26.548954 9400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:26.552762 9400 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:30:26.554731 9415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:26.555934 9400 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 11:30:26.556049 9400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "fb4a9e1560f84bf1a6eb1ac480f6e6f0"
format_stamp: "Formatted at 2026-09-26 11:30:25 on dist-test-slave-mrck"
server_key: "6d1ea0680aa15ea53d099e5d8f3e3be9"
server_key_iv: "9fd0186258c9fca473b7de98cad86397"
server_key_version: "encryptionkey@0"
I20260926 11:30:26.556564 9400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:26.569111 9400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:26.571837 9400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:26.572023 9400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:26.572600 9400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:26.573402 9400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:26.573467 9400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:26.573534 9400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:26.573580 9400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:26.582243 9400 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37875
I20260926 11:30:26.582263 9528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37875 every 8 connection(s)
I20260926 11:30:26.583232 9400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:30:26.584373 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 9400
Sep 26 11:30:26 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:26.594588 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:26.585427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42230 (local address 127.0.98.62:36565)
0926 11:30:26.585632 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:26.585635 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:26.585646 (+ 11us) server_negotiation.cc:407] Connection header received
0926 11:30:26.585687 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:26.585689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:26.585725 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:26.585809 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:26.586739 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.587651 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.588566 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.588899 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.589002 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:26.590348 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:26.590356 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:26.590371 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:26.590402 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:26.592349 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:26.592857 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:26.592867 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:26.592875 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:26.593015 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:26.593255 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:26.593258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:26.593260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:26.593664 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:26.593867 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:26.594141 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 11:30:26.594381 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":93,"threads_started":1}
Sep 26 11:30:26 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 11:30:26.595989 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:26.585051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46493 (local address 127.0.98.62:36565)
0926 11:30:26.585220 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:26.585224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:26.586255 (+ 1031us) server_negotiation.cc:407] Connection header received
0926 11:30:26.588017 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:26.588020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:26.588069 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:26.588161 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:26.589465 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.589943 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.591566 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.591848 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.591926 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:26.595913 (+ 3987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46493: BlockingRecv error: recv got EOF from 127.0.98.3:46493 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 11:30:26.596747 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:26.585644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.3:46493)
0926 11:30:26.586126 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:26.586165 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:26.587814 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:26.588333 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:26.588339 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:26.588770 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:26.589340 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:26.589354 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.590055 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.590058 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:26.591429 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:26.591443 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.591986 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.591991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:26.592197 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:26.592777 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:26.592808 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:26.595840 (+ 3032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":148,"threads_started":1}
W20260926 11:30:26.597199 9529 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36565 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:36565: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 11:30:26 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:26 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:26.601818 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:26.597594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.3:58275)
0926 11:30:26.597694 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:26.597709 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:26.597780 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:26.597960 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:26.597962 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:26.598075 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:26.598151 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:26.598155 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.598776 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.598778 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:26.599276 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:26.599282 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.599646 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.599649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:26.599714 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:26.600201 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:26.600219 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:26.601729 (+ 1510us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 11:30:26.601970 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:26.597657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58275 (local address 127.0.98.62:36565)
0926 11:30:26.597743 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:26.597746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:26.597758 (+ 12us) server_negotiation.cc:407] Connection header received
0926 11:30:26.597835 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:26.597837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:26.597865 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:26.597924 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:26.598248 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.598684 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.599359 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.599564 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.599635 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:26.601833 (+ 2198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58275: BlockingRecv error: recv got EOF from 127.0.98.3:58275 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 26 11:30:26 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:26 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:26.606344 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:26.602318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.3:43401)
0926 11:30:26.602426 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:26.602437 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:26.602492 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:26.602651 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:26.602653 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:26.602768 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:26.602825 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:26.602829 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.603424 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.603426 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:26.603955 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:26.603962 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.604332 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.604334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:26.604384 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:26.604807 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:26.604817 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:26.606267 (+ 1450us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 11:30:26.606472 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:26.602406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43401 (local address 127.0.98.62:36565)
0926 11:30:26.602516 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:26.602519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:26.602531 (+ 12us) server_negotiation.cc:407] Connection header received
0926 11:30:26.602556 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:26.602558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:26.602585 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:26.602639 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:26.602902 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.603347 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.604037 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:26.604234 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:26.604323 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:26.606368 (+ 2045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43401: BlockingRecv error: recv got EOF from 127.0.98.3:43401 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 11:30:26.606657 9529 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 11:30:27 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:27 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:27.247628 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:27.241892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.1:35919)
0926 11:30:27.242200 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:27.242215 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:27.242329 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:27.242585 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:27.242588 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:27.242777 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:27.242881 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:27.242887 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:27.243619 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:27.243622 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:27.244257 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:27.244266 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:27.244672 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:27.244675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:27.244757 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:27.245450 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:27.245463 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:27.247444 (+ 1981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":83,"threads_started":1}
W20260926 11:30:27.247691 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:27.242021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35919 (local address 127.0.98.62:36565)
0926 11:30:27.242234 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:27.242237 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:27.242287 (+ 50us) server_negotiation.cc:407] Connection header received
0926 11:30:27.242413 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:27.242415 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:27.242457 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:27.242541 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:27.242983 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:27.243510 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:27.244360 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:27.244576 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:27.244649 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:27.247566 (+ 2917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35919: BlockingRecv error: recv got EOF from 127.0.98.1:35919 (error 108)
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":69,"threads_started":1}
Sep 26 11:30:27 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:27 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:27.430632 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:27.424378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.2:44489)
0926 11:30:27.424644 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:27.424658 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:27.424756 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:27.425027 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:27.425029 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:27.425190 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:27.425300 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:27.425305 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:27.426490 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:27.426493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:27.427036 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:27.427043 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:27.427728 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:27.427731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:27.427885 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:27.428468 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:27.428479 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:27.430511 (+ 2032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":88,"threads_started":1}
W20260926 11:30:27.430871 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:27.424448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44489 (local address 127.0.98.62:36565)
0926 11:30:27.424617 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:27.424623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:27.424731 (+ 108us) server_negotiation.cc:407] Connection header received
0926 11:30:27.424834 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:27.424839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:27.424904 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:27.425012 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:27.425402 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:27.426385 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:27.427144 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:27.427599 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:27.427740 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:27.430712 (+ 2972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44489: BlockingRecv error: recv got EOF from 127.0.98.2:44489 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 11:30:27 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:27 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:27.612895 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:27.607371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.3:37679)
0926 11:30:27.607663 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:27.607680 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:27.607868 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:27.608137 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:27.608140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:27.608356 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:27.608448 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:27.608454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:27.609209 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:27.609214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:27.609813 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:27.609821 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:27.610276 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:27.610280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:27.610368 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:27.610905 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:27.610923 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:27.612753 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":82,"threads_started":1}
W20260926 11:30:27.613051 9535 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:27.607478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37679 (local address 127.0.98.62:36565)
0926 11:30:27.607647 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:27.607651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:27.607780 (+ 129us) server_negotiation.cc:407] Connection header received
0926 11:30:27.607980 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:27.607984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:27.608037 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:27.608119 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:27.608566 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:27.609097 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:27.609929 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:27.610164 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:27.610274 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:27.612894 (+ 2620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37679: BlockingRecv error: recv got EOF from 127.0.98.3:37679 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 26 11:30:28 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:28 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:28.253795 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:28.248408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.1:51911)
0926 11:30:28.248662 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:28.248675 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:28.248766 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:28.248991 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:28.248992 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:28.249165 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:28.249276 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:28.249282 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:28.249986 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:28.249988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:28.250549 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:28.250556 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:28.250961 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:28.250964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:28.251025 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:28.251497 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:28.251508 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:28.253648 (+ 2140us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":85,"threads_started":1}
W20260926 11:30:28.253931 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:28.248486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51911 (local address 127.0.98.62:36565)
0926 11:30:28.248736 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:28.248740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:28.248754 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:30:28.248835 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:28.248838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:28.248878 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:28.248961 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:28.249377 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:28.249900 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:28.250646 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:28.250869 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:28.250943 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:28.253785 (+ 2842us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51911: BlockingRecv error: recv got EOF from 127.0.98.1:51911 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":96,"threads_started":1}
Sep 26 11:30:28 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:28 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:28.436975 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:28.431351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.2:34775)
0926 11:30:28.431660 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:28.431677 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:28.431841 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:28.432095 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:28.432098 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:28.432311 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:28.432400 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:28.432406 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:28.433148 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:28.433155 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:28.433875 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:28.433885 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:28.434394 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:28.434397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:28.434502 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:28.435053 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:28.435067 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:28.436841 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":84,"threads_started":1}
W20260926 11:30:28.437109 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:28.431461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34775 (local address 127.0.98.62:36565)
0926 11:30:28.431628 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:28.431633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:28.431777 (+ 144us) server_negotiation.cc:407] Connection header received
0926 11:30:28.431938 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:28.431942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:28.431993 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:28.432076 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:28.432518 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:28.433047 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:28.433985 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:28.434276 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:28.434374 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:28.436970 (+ 2596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34775: BlockingRecv error: recv got EOF from 127.0.98.2:34775 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 26 11:30:28 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:28 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:28.619055 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:28.613671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.3:34761)
0926 11:30:28.613991 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:28.614005 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:28.614133 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:28.614390 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:28.614392 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:28.614592 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:28.614678 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:28.614684 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:28.615408 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:28.615413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:28.616041 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:28.616049 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:28.616473 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:28.616477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:28.616557 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:28.617071 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:28.617089 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:28.618865 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":96,"threads_started":1}
W20260926 11:30:28.619088 9539 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:28.613803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34761 (local address 127.0.98.62:36565)
0926 11:30:28.613941 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:28.613946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:28.614077 (+ 131us) server_negotiation.cc:407] Connection header received
0926 11:30:28.614229 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:28.614233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:28.614285 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:28.614366 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:28.614793 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:28.615305 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:28.616150 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:28.616370 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:28.616474 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:28.618989 (+ 2515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34761: BlockingRecv error: recv got EOF from 127.0.98.3:34761 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 11:30:29 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:29 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:29.266482 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:29.254651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.1:47759)
0926 11:30:29.254940 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:29.254958 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:29.255077 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:29.255457 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:29.255460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:29.255639 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:29.255823 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:29.255830 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:29.256714 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:29.256721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:29.257481 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:29.257491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:29.257936 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:29.257939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:29.257999 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:29.264173 (+ 6174us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:29.264189 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:29.266323 (+ 2134us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":99,"threads_started":1}
W20260926 11:30:29.266741 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:29.254802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47759 (local address 127.0.98.62:36565)
0926 11:30:29.255075 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:29.255079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:29.255095 (+ 16us) server_negotiation.cc:407] Connection header received
0926 11:30:29.255164 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:29.255166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:29.255208 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:29.255294 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:29.255968 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:29.256615 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:29.257590 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:29.257825 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:29.257923 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:29.266503 (+ 8580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47759: BlockingRecv error: recv got EOF from 127.0.98.1:47759 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":88,"threads_started":1}
Sep 26 11:30:29 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:29 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:29.443722 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:29.437882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.2:53767)
0926 11:30:29.438221 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:29.438236 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:29.438378 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:29.438652 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:29.438655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:29.438877 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:29.438964 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:29.438970 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:29.439775 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:29.439778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:29.440363 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:29.440371 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:29.440818 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:29.440821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:29.440905 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:29.441439 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:29.441451 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:29.443565 (+ 2114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":88,"threads_started":1}
W20260926 11:30:29.443912 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:29.438008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53767 (local address 127.0.98.62:36565)
0926 11:30:29.438195 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:29.438199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:29.438322 (+ 123us) server_negotiation.cc:407] Connection header received
0926 11:30:29.438488 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:29.438493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:29.438545 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:29.438630 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:29.439081 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:29.439631 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:29.440484 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:29.440708 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:29.440814 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:29.443776 (+ 2962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53767: BlockingRecv error: recv got EOF from 127.0.98.2:53767 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 26 11:30:29 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:29 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:29.624883 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:29.619842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.3:35771)
0926 11:30:29.620078 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:29.620099 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:29.620196 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:29.620400 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:29.620402 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:29.620568 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:29.620657 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:29.620673 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:29.621386 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:29.621388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:29.621942 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:29.621949 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:29.622350 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:29.622353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:29.622409 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:29.622962 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:29.622973 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:29.624749 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":85,"threads_started":1}
W20260926 11:30:29.625000 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:29.619939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35771 (local address 127.0.98.62:36565)
0926 11:30:29.620076 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:29.620080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:29.620146 (+ 66us) server_negotiation.cc:407] Connection header received
0926 11:30:29.620262 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:29.620264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:29.620308 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:29.620388 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:29.620765 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:29.621292 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:29.622035 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:29.622258 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:29.622333 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:29.624882 (+ 2549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35771: BlockingRecv error: recv got EOF from 127.0.98.3:35771 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 11:30:30 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:30 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:30.272496 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:30.267260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.1:36325)
0926 11:30:30.267529 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:30.267543 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:30.267636 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:30.267949 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:30.267951 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:30.268126 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:30.268226 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:30.268231 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:30.268939 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:30.268941 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:30.269493 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:30.269501 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:30.269951 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:30.269955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:30.270021 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:30.270537 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:30.270548 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:30.272381 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":67,"threads_started":1}
W20260926 11:30:30.272836 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:30.267365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36325 (local address 127.0.98.62:36565)
0926 11:30:30.267601 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:30.267605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:30.267618 (+ 13us) server_negotiation.cc:407] Connection header received
0926 11:30:30.267773 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:30.267775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:30.267822 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:30.267913 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:30.268328 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:30.268849 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:30.269595 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:30.269824 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:30.269903 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:30.272542 (+ 2639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36325: BlockingRecv error: recv got EOF from 127.0.98.1:36325 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":72,"threads_started":1}
Sep 26 11:30:30 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:30 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:30.449795 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:30.444657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.2:37097)
0926 11:30:30.444938 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:30.444953 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:30.445051 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:30.445272 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:30.445274 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:30.445445 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:30.445536 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:30.445541 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:30.446368 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:30.446371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:30.446946 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:30.446954 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:30.447383 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:30.447386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:30.447445 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:30.447956 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:30.447967 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:30.449683 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":85,"threads_started":1}
W20260926 11:30:30.450035 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:30.444911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37097 (local address 127.0.98.62:36565)
0926 11:30:30.445027 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:30.445030 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:30.445044 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:30:30.445119 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:30.445122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:30.445167 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:30.445243 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:30.445657 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:30.446286 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:30.447041 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:30.447255 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:30.447340 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:30.449898 (+ 2558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37097: BlockingRecv error: recv got EOF from 127.0.98.2:37097 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 26 11:30:30 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:30 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:30.631805 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:30.625856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.3:52035)
0926 11:30:30.626171 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:30.626188 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:30.626351 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:30.626668 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:30.626672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:30.626894 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:30.626993 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:30.626999 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:30.627955 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:30.627961 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:30.628729 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:30.628740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:30.629236 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:30.629239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:30.629303 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:30.629873 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:30.629887 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:30.631630 (+ 1743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":87,"threads_started":1}
W20260926 11:30:30.632040 9547 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:30.625929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52035 (local address 127.0.98.62:36565)
0926 11:30:30.626134 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:30.626139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:30.626287 (+ 148us) server_negotiation.cc:407] Connection header received
0926 11:30:30.626472 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:30.626476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:30.626535 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:30.626644 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:30.627150 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:30.627837 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:30.628882 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:30.629116 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:30.629198 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:30.631822 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52035: BlockingRecv error: recv got EOF from 127.0.98.3:52035 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 26 11:30:31 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:31 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:31.279198 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:31.273223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.1:43227)
0926 11:30:31.273504 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:31.273517 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:31.273599 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:31.273828 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:31.273830 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:31.274058 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:31.274177 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:31.274183 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:31.274992 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:31.274995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:31.275655 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:31.275664 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:31.276153 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:31.276157 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:31.276260 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:31.276829 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:31.276851 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:31.279027 (+ 2176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":80,"threads_started":1}
W20260926 11:30:31.279284 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:31.273331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43227 (local address 127.0.98.62:36565)
0926 11:30:31.273572 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:31.273576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:31.273593 (+ 17us) server_negotiation.cc:407] Connection header received
0926 11:30:31.273687 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:31.273689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:31.273730 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:31.273808 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:31.274282 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:31.274900 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:31.275798 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:31.276024 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:31.276110 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:31.279149 (+ 3039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43227: BlockingRecv error: recv got EOF from 127.0.98.1:43227 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":54,"threads_started":1}
Sep 26 11:30:31 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:31 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:31.456214 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:31.450612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.2:42707)
0926 11:30:31.450883 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:31.450899 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:31.451008 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:31.451320 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:31.451322 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:31.451487 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:31.451578 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:31.451583 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:31.452779 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:31.452782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:31.453342 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:31.453350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:31.453896 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:31.453899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:31.453959 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:31.454378 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:31.454389 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:31.456100 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":96,"threads_started":1}
W20260926 11:30:31.456422 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:31.450716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42707 (local address 127.0.98.62:36565)
0926 11:30:31.450869 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:31.450874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:31.450968 (+ 94us) server_negotiation.cc:407] Connection header received
0926 11:30:31.451133 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:31.451136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:31.451201 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:31.451302 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:31.451694 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:31.452651 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:31.453455 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:31.453793 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:31.453876 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:31.456295 (+ 2419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42707: BlockingRecv error: recv got EOF from 127.0.98.2:42707 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 11:30:31 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:31 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:31.638453 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:31.632710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.3:34383)
0926 11:30:31.632989 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:31.633003 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:31.633099 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:31.633351 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:31.633352 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:31.633526 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:31.633616 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:31.633621 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:31.634398 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:31.634401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:31.635037 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:31.635048 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:31.635630 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:31.635634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:31.635719 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:31.636485 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:31.636500 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:31.638338 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":108,"threads_started":1}
W20260926 11:30:31.638679 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:31.632785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34383 (local address 127.0.98.62:36565)
0926 11:30:31.632942 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:31.632946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:31.633101 (+ 155us) server_negotiation.cc:407] Connection header received
0926 11:30:31.633182 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:31.633184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:31.633232 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:31.633315 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:31.633743 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:31.634293 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:31.635194 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:31.635496 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:31.635613 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:31.638545 (+ 2932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34383: BlockingRecv error: recv got EOF from 127.0.98.3:34383 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 26 11:30:32 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:32 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:32.285805 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:32.280071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.1:45821)
0926 11:30:32.280362 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:32.280377 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:32.280472 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:32.280808 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:32.280810 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:32.280987 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:32.281084 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:32.281089 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:32.281906 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:32.281909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:32.282546 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:32.282556 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:32.283160 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:32.283171 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:32.283258 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:32.283861 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:32.283873 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:32.285691 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":109,"threads_started":1}
W20260926 11:30:32.286031 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:32.280168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45821 (local address 127.0.98.62:36565)
0926 11:30:32.280413 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:32.280418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:32.280432 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:30:32.280614 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:32.280617 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:32.280671 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:32.280779 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:32.281197 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:32.281778 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:32.282666 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:32.283011 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:32.283128 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:32.285876 (+ 2748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45821: BlockingRecv error: recv got EOF from 127.0.98.1:45821 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":101,"threads_started":1}
Sep 26 11:30:32 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:32 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:32.462745 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:32.457004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.2:57907)
0926 11:30:32.457301 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:32.457317 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:32.457476 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:32.457739 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:32.457742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:32.457947 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:32.458034 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:32.458039 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:32.458801 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:32.458807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:32.459447 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:32.459457 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:32.460125 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:32.460129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:32.460233 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:32.460875 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:32.460886 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:32.462625 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":84,"threads_started":1}
W20260926 11:30:32.462875 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:32.457095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57907 (local address 127.0.98.62:36565)
0926 11:30:32.457275 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:32.457280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:32.457411 (+ 131us) server_negotiation.cc:407] Connection header received
0926 11:30:32.457577 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:32.457580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:32.457632 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:32.457721 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:32.458165 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:32.458693 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:32.459590 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:32.459977 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:32.460091 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:32.462760 (+ 2669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57907: BlockingRecv error: recv got EOF from 127.0.98.2:57907 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 26 11:30:32 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:32 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:32.644863 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:32.639286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.3:50689)
0926 11:30:32.639579 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:32.639594 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:32.639703 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:32.639975 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:32.639977 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:32.640175 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:32.640295 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:32.640300 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:32.641047 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:32.641050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:32.641665 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:32.641673 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:32.642067 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:32.642069 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:32.642132 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:32.642743 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:32.642756 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:32.644749 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":68,"threads_started":1}
W20260926 11:30:32.645066 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:32.639372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50689 (local address 127.0.98.62:36565)
0926 11:30:32.639558 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:32.639563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:32.639672 (+ 109us) server_negotiation.cc:407] Connection header received
0926 11:30:32.639810 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:32.639813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:32.639863 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:32.639930 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:32.640402 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:32.640952 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:32.641767 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:32.641972 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:32.642049 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:32.644916 (+ 2867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50689: BlockingRecv error: recv got EOF from 127.0.98.3:50689 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 26 11:30:33 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:33 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:33.292013 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:33.286576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.1:43525)
0926 11:30:33.286906 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:33.286919 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:33.287009 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:33.287284 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:33.287286 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:33.287465 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:33.287560 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:33.287565 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:33.288373 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:33.288375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:33.288929 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:33.288937 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:33.289404 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:33.289406 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:33.289462 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:33.289978 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:33.289990 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:33.291897 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":84,"threads_started":1}
W20260926 11:30:33.292198 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:33.286678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43525 (local address 127.0.98.62:36565)
0926 11:30:33.286944 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:33.286948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:33.286967 (+ 19us) server_negotiation.cc:407] Connection header received
0926 11:30:33.287130 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:33.287133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:33.287175 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:33.287251 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:33.287655 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:33.288271 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:33.289049 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:33.289300 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:33.289392 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:33.292047 (+ 2655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:43525: BlockingRecv error: recv got EOF from 127.0.98.1:43525 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":92,"threads_started":1}
Sep 26 11:30:33 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:33 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:33.469357 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:33.463642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.2:35795)
0926 11:30:33.463966 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:33.463983 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:33.464098 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:33.464354 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:33.464356 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:33.464597 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:33.464699 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:33.464705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:33.465500 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:33.465503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:33.466155 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:33.466179 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:33.466643 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:33.466656 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:33.466743 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:33.467227 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:33.467238 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:33.469231 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":122,"threads_started":1}
W20260926 11:30:33.469537 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:33.463799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35795 (local address 127.0.98.62:36565)
0926 11:30:33.463943 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:33.463946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:33.464059 (+ 113us) server_negotiation.cc:407] Connection header received
0926 11:30:33.464193 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:33.464196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:33.464244 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:33.464317 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:33.464848 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:33.465386 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:33.466298 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:33.466515 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:33.466602 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:33.469377 (+ 2775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35795: BlockingRecv error: recv got EOF from 127.0.98.2:35795 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 11:30:33 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:33 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:33.651985 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:33.645699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.3:41617)
0926 11:30:33.646095 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:33.646109 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:33.646324 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:33.646678 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:33.646680 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:33.646893 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:33.647012 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:33.647022 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:33.648087 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:33.648089 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:33.648742 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:33.648751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:33.649202 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:33.649205 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:33.649273 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:33.649860 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:33.649873 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:33.651869 (+ 1996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":104,"threads_started":1}
W20260926 11:30:33.652251 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:33.645829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41617 (local address 127.0.98.62:36565)
0926 11:30:33.646009 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:33.646014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:33.646235 (+ 221us) server_negotiation.cc:407] Connection header received
0926 11:30:33.646485 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:33.646488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:33.646555 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:33.646650 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:33.647158 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:33.647962 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:33.648844 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:33.649053 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:33.649134 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:33.652018 (+ 2884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41617: BlockingRecv error: recv got EOF from 127.0.98.3:41617 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 26 11:30:34 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:34 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:34.302333 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:34.292878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.1:57653)
0926 11:30:34.293221 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:34.293240 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:34.293354 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:34.293645 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:34.293648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:34.293837 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:34.293942 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:34.293948 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:34.294716 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:34.294719 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:34.295278 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:34.295286 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:34.295830 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:34.295841 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:34.295914 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:34.299791 (+ 3877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:34.299814 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:34.302159 (+ 2345us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":91,"threads_started":1}
W20260926 11:30:34.302426 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:34.293092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57653 (local address 127.0.98.62:36565)
0926 11:30:34.293363 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:34.293366 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:34.293380 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:30:34.293421 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:34.293423 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:34.293466 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:34.293577 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:34.294061 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:34.294600 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:34.295381 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:34.295613 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:34.295695 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:34.302296 (+ 6601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57653: BlockingRecv error: recv got EOF from 127.0.98.1:57653 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":107,"threads_started":1}
Sep 26 11:30:34 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:34 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:34.475976 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:34.470130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.2:57353)
0926 11:30:34.470399 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:34.470412 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:34.470509 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:34.470731 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:34.470733 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:34.470907 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:34.471000 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:34.471004 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:34.471902 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:34.471904 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:34.472454 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:34.472461 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:34.472851 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:34.472854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:34.472909 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:34.473456 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:34.473467 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:34.475811 (+ 2344us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":79,"threads_started":1}
W20260926 11:30:34.476228 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:34.470204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57353 (local address 127.0.98.62:36565)
0926 11:30:34.470334 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:34.470338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:34.470477 (+ 139us) server_negotiation.cc:407] Connection header received
0926 11:30:34.470575 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:34.470578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:34.470619 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:34.470694 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:34.471093 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:34.471795 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:34.472542 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:34.472759 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:34.472836 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:34.475982 (+ 3146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57353: BlockingRecv error: recv got EOF from 127.0.98.2:57353 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 26 11:30:34 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:34 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:34.658183 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:34.652800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.3:56735)
0926 11:30:34.653192 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:34.653212 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:34.653348 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:34.653619 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:34.653622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:34.653822 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:34.653913 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:34.653919 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:34.654657 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:34.654663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:34.655224 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:34.655231 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:34.655684 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:34.655688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:34.655807 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:34.656290 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:34.656301 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:34.658065 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":77,"threads_started":1}
W20260926 11:30:34.658342 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:34.652874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56735 (local address 127.0.98.62:36565)
0926 11:30:34.653024 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:34.653030 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:34.653289 (+ 259us) server_negotiation.cc:407] Connection header received
0926 11:30:34.653448 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:34.653453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:34.653502 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:34.653596 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:34.654030 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:34.654551 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:34.655348 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:34.655577 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:34.655685 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:34.658190 (+ 2505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56735: BlockingRecv error: recv got EOF from 127.0.98.3:56735 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 11:30:35 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:35 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:35.308497 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:35.303118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.1:42563)
0926 11:30:35.303419 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:35.303432 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:35.303516 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:35.303820 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:35.303822 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:35.303997 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:35.304094 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:35.304099 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:35.304825 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:35.304828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:35.305391 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:35.305398 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:35.305806 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:35.305809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:35.305869 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:35.306352 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:35.306364 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:35.308377 (+ 2013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":92,"threads_started":1}
W20260926 11:30:35.308703 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:35.303191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42563 (local address 127.0.98.62:36565)
0926 11:30:35.303455 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:35.303459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:35.303493 (+ 34us) server_negotiation.cc:407] Connection header received
0926 11:30:35.303606 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:35.303608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:35.303651 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:35.303780 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:35.304208 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:35.304732 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:35.305485 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:35.305709 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:35.305787 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:35.308527 (+ 2740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42563: BlockingRecv error: recv got EOF from 127.0.98.1:42563 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":117,"threads_started":1}
Sep 26 11:30:35 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:35 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:35.482167 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:35.476693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.2:32885)
0926 11:30:35.477008 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:35.477025 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:35.477141 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:35.477412 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:35.477415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:35.477644 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:35.477742 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:35.477747 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:35.478594 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:35.478597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:35.479156 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:35.479163 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:35.479592 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:35.479595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:35.479653 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:35.480237 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:35.480258 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:35.482051 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":92,"threads_started":1}
W20260926 11:30:35.482424 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:35.476727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32885 (local address 127.0.98.62:36565)
0926 11:30:35.476920 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:35.476925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:35.477128 (+ 203us) server_negotiation.cc:407] Connection header received
0926 11:30:35.477223 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:35.477229 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:35.477284 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:35.477358 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:35.477875 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:35.478460 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:35.479276 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:35.479494 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:35.479581 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:35.482310 (+ 2729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:32885: BlockingRecv error: recv got EOF from 127.0.98.2:32885 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 11:30:35 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:35 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:35.664726 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:35.658985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.3:56045)
0926 11:30:35.659282 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:35.659297 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:35.659407 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:35.659646 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:35.659648 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:35.659914 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:35.660031 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:35.660037 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:35.660794 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:35.660797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:35.661480 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:35.661490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:35.661942 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:35.661945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:35.662013 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:35.662634 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:35.662646 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:35.664546 (+ 1900us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":111,"threads_started":1}
W20260926 11:30:35.664767 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:35.659112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56045 (local address 127.0.98.62:36565)
0926 11:30:35.659266 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:35.659270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:35.659354 (+ 84us) server_negotiation.cc:407] Connection header received
0926 11:30:35.659480 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:35.659482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:35.659525 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:35.659617 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:35.660144 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:35.660663 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:35.661602 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:35.661833 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:35.661912 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:35.664667 (+ 2755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56045: BlockingRecv error: recv got EOF from 127.0.98.3:56045 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 11:30:36 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:36 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:36.314920 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:36.309292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.1:50015)
0926 11:30:36.309557 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:36.309571 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:36.309668 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:36.309913 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:36.309915 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:36.310090 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:36.310187 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:36.310192 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:36.310997 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:36.311000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:36.311569 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:36.311578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:36.312139 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:36.312142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:36.312239 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:36.312829 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:36.312841 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:36.314799 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":87,"threads_started":1}
W20260926 11:30:36.315117 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:36.309341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50015 (local address 127.0.98.62:36565)
0926 11:30:36.309580 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:36.309583 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:36.309652 (+ 69us) server_negotiation.cc:407] Connection header received
0926 11:30:36.309759 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:36.309761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:36.309802 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:36.309882 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:36.310312 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:36.310864 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:36.311693 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:36.311991 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:36.312073 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:36.314952 (+ 2879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50015: BlockingRecv error: recv got EOF from 127.0.98.1:50015 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":58,"threads_started":1}
Sep 26 11:30:36 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:36 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:36.488533 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:36.482969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.2:53567)
0926 11:30:36.483265 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:36.483279 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:36.483425 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:36.483696 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:36.483698 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:36.483955 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:36.484058 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:36.484064 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:36.484892 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:36.484894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:36.485509 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:36.485518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:36.485948 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:36.485951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:36.486011 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:36.486509 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:36.486520 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:36.488416 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":96,"threads_started":1}
W20260926 11:30:36.488746 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:36.483044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53567 (local address 127.0.98.62:36565)
0926 11:30:36.483218 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:36.483222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:36.483415 (+ 193us) server_negotiation.cc:407] Connection header received
0926 11:30:36.483548 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:36.483551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:36.483596 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:36.483665 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:36.484225 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:36.484789 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:36.485632 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:36.485854 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:36.485935 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:36.488572 (+ 2637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53567: BlockingRecv error: recv got EOF from 127.0.98.2:53567 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 26 11:30:36 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:36 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:36.671346 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:36.665612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.3:60077)
0926 11:30:36.665955 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:36.665969 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:36.666068 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:36.666386 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:36.666389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:36.666569 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:36.666663 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:36.666668 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:36.667375 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:36.667377 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:36.667976 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:36.667983 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:36.668447 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:36.668450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:36.668514 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:36.668994 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:36.669008 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:36.671223 (+ 2215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":111,"threads_started":1}
W20260926 11:30:36.671471 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:36.665809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60077 (local address 127.0.98.62:36565)
0926 11:30:36.665952 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:36.665955 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:36.666041 (+ 86us) server_negotiation.cc:407] Connection header received
0926 11:30:36.666191 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:36.666194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:36.666241 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:36.666319 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:36.666769 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:36.667284 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:36.668113 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:36.668338 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:36.668426 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:36.671336 (+ 2910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60077: BlockingRecv error: recv got EOF from 127.0.98.3:60077 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 11:30:37 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:37 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:37.321920 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:37.315778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.1:46489)
0926 11:30:37.316054 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:37.316067 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:37.316161 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:37.316440 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:37.316443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:37.316672 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:37.316789 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:37.316796 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:37.317809 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:37.317812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:37.318458 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:37.318469 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:37.318912 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:37.318915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:37.318985 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:37.319514 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:37.319529 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:37.321758 (+ 2229us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":99,"threads_started":1}
W20260926 11:30:37.322021 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:37.315858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46489 (local address 127.0.98.62:36565)
0926 11:30:37.316091 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:37.316094 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:37.316110 (+ 16us) server_negotiation.cc:407] Connection header received
0926 11:30:37.316231 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:37.316234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:37.316276 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:37.316353 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:37.316920 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:37.317660 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:37.318580 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:37.318804 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:37.318881 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:37.321908 (+ 3027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46489: BlockingRecv error: recv got EOF from 127.0.98.1:46489 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":81,"threads_started":1}
Sep 26 11:30:37 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:37 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:37.494887 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:37.489418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.2:42677)
0926 11:30:37.489705 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:37.489725 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:37.489866 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:37.490142 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:37.490145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:37.490358 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:37.490447 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:37.490453 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:37.491220 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:37.491224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:37.491852 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:37.491861 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:37.492322 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:37.492326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:37.492413 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:37.492974 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:37.492994 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:37.494703 (+ 1709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":83,"threads_started":1}
W20260926 11:30:37.494944 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:37.489478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42677 (local address 127.0.98.62:36565)
0926 11:30:37.489652 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:37.489656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:37.489803 (+ 147us) server_negotiation.cc:407] Connection header received
0926 11:30:37.489966 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:37.489970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:37.490026 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:37.490111 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:37.490564 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:37.491096 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:37.491966 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:37.492202 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:37.492310 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:37.494833 (+ 2523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42677: BlockingRecv error: recv got EOF from 127.0.98.2:42677 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 26 11:30:37 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:37 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:37.678068 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:37.672211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.3:52325)
0926 11:30:37.672477 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:37.672489 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:37.672592 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:37.672826 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:37.672828 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:37.673016 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:37.673112 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:37.673117 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:37.674031 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:37.674037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:37.674714 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:37.674726 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:37.675161 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:37.675164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:37.675232 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:37.675931 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:37.675944 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:37.677908 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":81,"threads_started":1}
W20260926 11:30:37.678279 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:37.672274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52325 (local address 127.0.98.62:36565)
0926 11:30:37.672414 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:37.672417 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:37.672557 (+ 140us) server_negotiation.cc:407] Connection header received
0926 11:30:37.672672 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:37.672674 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:37.672719 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:37.672794 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:37.673226 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:37.673893 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:37.674831 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:37.675045 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:37.675125 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:37.678112 (+ 2987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52325: BlockingRecv error: recv got EOF from 127.0.98.3:52325 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 11:30:38 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:38 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:38.328891 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:38.323024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.1:35353)
0926 11:30:38.323299 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:38.323313 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:38.323407 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:38.323648 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:38.323650 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:38.323853 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:38.323948 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:38.323954 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:38.324669 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:38.324673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:38.325248 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:38.325256 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:38.325915 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:38.325919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:38.326010 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:38.326650 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:38.326665 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:38.328775 (+ 2110us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":98,"threads_started":1}
W20260926 11:30:38.329128 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:38.323095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35353 (local address 127.0.98.62:36565)
0926 11:30:38.323359 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:38.323362 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:38.323378 (+ 16us) server_negotiation.cc:407] Connection header received
0926 11:30:38.323499 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:38.323501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:38.323545 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:38.323634 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:38.324051 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:38.324565 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:38.325394 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:38.325745 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:38.325911 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:38.328969 (+ 3058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35353: BlockingRecv error: recv got EOF from 127.0.98.1:35353 (error 108)
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":114,"threads_started":1}
Sep 26 11:30:38 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:38 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:38.501116 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:38.495805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.2:46901)
0926 11:30:38.496077 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:38.496090 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:38.496187 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:38.496415 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:38.496418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:38.496593 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:38.496700 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:38.496705 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:38.497410 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:38.497412 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:38.497965 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:38.497972 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:38.498405 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:38.498409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:38.498472 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:38.499061 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:38.499073 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:38.500991 (+ 1918us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":103,"threads_started":1}
W20260926 11:30:38.501317 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:38.495889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46901 (local address 127.0.98.62:36565)
0926 11:30:38.496037 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:38.496040 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:38.496145 (+ 105us) server_negotiation.cc:407] Connection header received
0926 11:30:38.496260 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:38.496262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:38.496303 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:38.496376 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:38.496801 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:38.497319 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:38.498064 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:38.498282 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:38.498362 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:38.501167 (+ 2805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46901: BlockingRecv error: recv got EOF from 127.0.98.2:46901 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 11:30:38 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:38 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:38.684794 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:38.679037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.3:33897)
0926 11:30:38.679321 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:38.679334 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:38.679468 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:38.679784 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:38.679787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:38.679978 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:38.680075 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:38.680080 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:38.680853 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:38.680856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:38.681449 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:38.681458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:38.681941 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:38.681944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:38.682018 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:38.682685 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:38.682709 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:38.684679 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":85,"threads_started":1}
W20260926 11:30:38.684991 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:38.679084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33897 (local address 127.0.98.62:36565)
0926 11:30:38.679251 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:38.679255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:38.679455 (+ 200us) server_negotiation.cc:407] Connection header received
0926 11:30:38.679595 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:38.679597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:38.679645 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:38.679713 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:38.680179 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:38.680690 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:38.681562 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:38.681792 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:38.681882 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:38.684828 (+ 2946us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33897: BlockingRecv error: recv got EOF from 127.0.98.3:33897 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 11:30:39 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:39 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:39.340232 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:39.329685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.1:44213)
0926 11:30:39.329968 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:39.329981 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:39.330070 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:39.330320 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:39.330323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:39.330502 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:39.330599 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:39.330603 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:39.331363 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:39.331366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:39.332027 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:39.332037 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:39.332714 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:39.332717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:39.332791 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:39.338136 (+ 5345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:39.338165 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:39.340103 (+ 1938us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":105,"threads_started":1}
W20260926 11:30:39.340500 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:39.329793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44213 (local address 127.0.98.62:36565)
0926 11:30:39.330050 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:39.330053 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:39.330069 (+ 16us) server_negotiation.cc:407] Connection header received
0926 11:30:39.330152 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:39.330154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:39.330196 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:39.330279 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:39.330716 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:39.331252 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:39.332304 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:39.332566 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:39.332667 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:39.340335 (+ 7668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44213: BlockingRecv error: recv got EOF from 127.0.98.1:44213 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":99,"threads_started":1}
Sep 26 11:30:39 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:39 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:39.507400 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:39.502023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.2:37009)
0926 11:30:39.502396 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:39.502419 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:39.502552 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:39.502857 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:39.502860 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:39.503110 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:39.503232 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:39.503240 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:39.503997 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:39.503999 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:39.504542 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:39.504549 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:39.504987 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:39.504989 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:39.505048 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:39.505506 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:39.505516 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:39.507286 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":110,"threads_started":1}
W20260926 11:30:39.507668 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:39.502178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37009 (local address 127.0.98.62:36565)
0926 11:30:39.502350 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:39.502355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:39.502503 (+ 148us) server_negotiation.cc:407] Connection header received
0926 11:30:39.502661 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:39.502666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:39.502749 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:39.502832 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:39.503347 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:39.503899 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:39.504674 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:39.504890 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:39.504977 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:39.507431 (+ 2454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37009: BlockingRecv error: recv got EOF from 127.0.98.2:37009 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 11:30:39 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:39 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:39.690865 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:39.685641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.3:49025)
0926 11:30:39.685917 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:39.685930 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:39.686039 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:39.686314 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:39.686316 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:39.686493 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:39.686588 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:39.686593 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:39.687312 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:39.687314 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:39.687962 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:39.687972 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:39.688394 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:39.688397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:39.688460 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:39.688942 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:39.688954 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:39.690747 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":96,"threads_started":1}
W20260926 11:30:39.691061 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:39.685697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49025 (local address 127.0.98.62:36565)
0926 11:30:39.685846 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:39.685849 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:39.686005 (+ 156us) server_negotiation.cc:407] Connection header received
0926 11:30:39.686142 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:39.686145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:39.686191 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:39.686278 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:39.686687 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:39.687208 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:39.688067 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:39.688292 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:39.688374 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:39.690921 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49025: BlockingRecv error: recv got EOF from 127.0.98.3:49025 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 26 11:30:40 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:40 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:40.346508 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:40.340968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.1:37433)
0926 11:30:40.341311 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:40.341324 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:40.341419 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:40.341630 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:40.341632 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:40.341807 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:40.341898 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:40.341903 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:40.342740 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:40.342742 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:40.343308 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:40.343322 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:40.343856 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:40.343859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:40.343924 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:40.344459 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:40.344470 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:40.346353 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":90,"threads_started":1}
W20260926 11:30:40.346583 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:40.341119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37433 (local address 127.0.98.62:36565)
0926 11:30:40.341423 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:40.341427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:40.341440 (+ 13us) server_negotiation.cc:407] Connection header received
0926 11:30:40.341478 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:40.341480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:40.341519 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:40.341595 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:40.342038 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:40.342639 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:40.343449 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:40.343678 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:40.343854 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:40.346480 (+ 2626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37433: BlockingRecv error: recv got EOF from 127.0.98.1:37433 (error 108)
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":96,"threads_started":1}
Sep 26 11:30:40 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:40 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:40.513269 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:40.508226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.2:36397)
0926 11:30:40.508492 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:40.508505 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:40.508615 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:40.508818 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:40.508820 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:40.508995 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:40.509112 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:40.509117 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:40.509831 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:40.509833 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:40.510401 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:40.510408 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:40.510793 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:40.510795 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:40.510863 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:40.511359 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:40.511371 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:40.513156 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":100,"threads_started":1}
W20260926 11:30:40.513408 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:40.508285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36397 (local address 127.0.98.62:36565)
0926 11:30:40.508421 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:40.508424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:40.508571 (+ 147us) server_negotiation.cc:407] Connection header received
0926 11:30:40.508693 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:40.508695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:40.508738 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:40.508806 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:40.509205 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:40.509735 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:40.510498 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:40.510700 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:40.510774 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:40.513287 (+ 2513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36397: BlockingRecv error: recv got EOF from 127.0.98.2:36397 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 11:30:40 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:40 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:40.697392 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:40.691707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.3:51211)
0926 11:30:40.692058 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:40.692072 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:40.692199 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:40.692478 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:40.692481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:40.692686 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:40.692777 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:40.692783 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:40.693547 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:40.693552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:40.694144 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:40.694152 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:40.694609 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:40.694613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:40.694696 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:40.695259 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:40.695270 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:40.697251 (+ 1981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":109,"threads_started":1}
W20260926 11:30:40.697638 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:40.691835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51211 (local address 127.0.98.62:36565)
0926 11:30:40.692004 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:40.692008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:40.692146 (+ 138us) server_negotiation.cc:407] Connection header received
0926 11:30:40.692310 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:40.692314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:40.692369 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:40.692456 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:40.692892 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:40.693437 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:40.694268 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:40.694497 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:40.694606 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:40.697395 (+ 2789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51211: BlockingRecv error: recv got EOF from 127.0.98.3:51211 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 26 11:30:41 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:41 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:41.353201 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:41.347531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.1:41621)
0926 11:30:41.347892 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:41.347909 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:41.348033 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:41.348410 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:41.348414 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:41.348615 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:41.348717 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:41.348722 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:41.349490 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:41.349493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:41.350069 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:41.350078 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:41.350493 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:41.350496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:41.350558 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:41.351173 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:41.351185 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:41.353064 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":109,"threads_started":1}
W20260926 11:30:41.353435 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:41.347604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41621 (local address 127.0.98.62:36565)
0926 11:30:41.347908 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:41.347911 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:41.347971 (+ 60us) server_negotiation.cc:407] Connection header received
0926 11:30:41.348186 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:41.348189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:41.348241 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:41.348357 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:41.348833 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:41.349372 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:41.350184 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:41.350398 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:41.350472 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:41.353258 (+ 2786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41621: BlockingRecv error: recv got EOF from 127.0.98.1:41621 (error 108)
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":70,"threads_started":1}
Sep 26 11:30:41 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 11:30:41 dist-test-slave-mrck krb5kdc[8533](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 11:30:41.519945 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:41.514227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36565 (local address 127.0.98.2:35543)
0926 11:30:41.514511 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:41.514525 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:41.514631 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:41.514921 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:41.514923 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:41.515108 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:41.515208 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:41.515213 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:41.515980 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:41.515982 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:41.516553 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:41.516560 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:41.517144 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:41.517148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:41.517249 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:41.517778 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:41.517789 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:41.519801 (+ 2012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:36565: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":83,"threads_started":1}
W20260926 11:30:41.520076 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:41.514374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35543 (local address 127.0.98.62:36565)
0926 11:30:41.514557 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:41.514561 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:41.514575 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:30:41.514731 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:41.514734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:41.514776 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:41.514846 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:41.515326 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:41.515886 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:41.516702 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:41.516933 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:41.517115 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:41.519929 (+ 2814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35543: BlockingRecv error: recv got EOF from 127.0.98.2:35543 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 11:30:41.594154 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 9116
I20260926 11:30:41.599622 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 9267
I20260926 11:30:41.605284 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 9400
I20260926 11:30:41.610471 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 9046
2026-09-26T11:30:41Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17959 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T11:30:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:30:41Z Disabled control of system clock
I20260926 11:30:41.640442 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38961
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38961
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 11:30:41.747968 9596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:41.748227 9596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:41.748262 9596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:41.752398 9596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:30:41.752455 9596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:41.752471 9596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:41.752488 9596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:30:41.752503 9596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:30:41.757776 9596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38961
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:41.758831 9596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:41.760272 9596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:41.768814 9604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:41.768872 9602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:41.768870 9596 server_base.cc:1034] running on GCE node
W20260926 11:30:41.768836 9601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:41.769583 9596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:41.770671 9596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:41.771807 9596 hybrid_clock.cc:648] HybridClock initialized: now 1790422241771794 us; error 27 us; skew 500 ppm
I20260926 11:30:41.773507 9596 webserver.cc:460] Webserver started at http://127.0.98.62:35063/ using document root <none> and password file <none>
I20260926 11:30:41.774077 9596 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:41.774164 9596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:41.774387 9596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:41.776505 9596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "d785596b4532417eacf100bdfcc4025d"
format_stamp: "Formatted at 2026-09-26 11:30:41 on dist-test-slave-mrck"
server_key: "7d306b31908c0d91aa16862ea0a6a96e"
server_key_iv: "29d03991b4b015083c7478cb608f2312"
server_key_version: "encryptionkey@0"
I20260926 11:30:41.777062 9596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "d785596b4532417eacf100bdfcc4025d"
format_stamp: "Formatted at 2026-09-26 11:30:41 on dist-test-slave-mrck"
server_key: "7d306b31908c0d91aa16862ea0a6a96e"
server_key_iv: "29d03991b4b015083c7478cb608f2312"
server_key_version: "encryptionkey@0"
I20260926 11:30:41.780473 9596 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:30:41.782780 9610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:41.784039 9596 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:30:41.784173 9596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "d785596b4532417eacf100bdfcc4025d"
format_stamp: "Formatted at 2026-09-26 11:30:41 on dist-test-slave-mrck"
server_key: "7d306b31908c0d91aa16862ea0a6a96e"
server_key_iv: "29d03991b4b015083c7478cb608f2312"
server_key_version: "encryptionkey@0"
I20260926 11:30:41.784282 9596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:41.796968 9596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:41.797597 9596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:41.797792 9596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:41.804737 9596 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38961
I20260926 11:30:41.804738 9662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38961 every 8 connection(s)
I20260926 11:30:41.805665 9596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:30:41.808648 9663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:41.814220 9663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d785596b4532417eacf100bdfcc4025d: Bootstrap starting.
I20260926 11:30:41.815779 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 9596
I20260926 11:30:41.815908 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:30:41.816207 392 external_mini_cluster.cc:1428] Setting key 571a411bbaa627bb803cac048a8c8344
I20260926 11:30:41.816947 9663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d785596b4532417eacf100bdfcc4025d: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:41.817880 9663 log.cc:824] T 00000000000000000000000000000000 P d785596b4532417eacf100bdfcc4025d: Log is configured to *not* fsync() on all Append() calls
I20260926 11:30:41.819954 9663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d785596b4532417eacf100bdfcc4025d: No bootstrap required, opened a new log
I20260926 11:30:41.824077 9663 raft_consensus.cc:348] T 00000000000000000000000000000000 P d785596b4532417eacf100bdfcc4025d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d785596b4532417eacf100bdfcc4025d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38961 } }
I20260926 11:30:41.824344 9663 raft_consensus.cc:374] T 00000000000000000000000000000000 P d785596b4532417eacf100bdfcc4025d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:41.824443 9663 raft_consensus.cc:729] T 00000000000000000000000000000000 P d785596b4532417eacf100bdfcc4025d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d785596b4532417eacf100bdfcc4025d, State: Initialized, Role: FOLLOWER
I20260926 11:30:41.824982 9663 consensus_queue.cc:260] T 00000000000000000000000000000000 P d785596b4532417eacf100bdfcc4025d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d785596b4532417eacf100bdfcc4025d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38961 } }
I20260926 11:30:41.825112 9663 raft_consensus.cc:388] T 00000000000000000000000000000000 P d785596b4532417eacf100bdfcc4025d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:30:41.825193 9663 raft_consensus.cc:482] T 00000000000000000000000000000000 P d785596b4532417eacf100bdfcc4025d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:30:41.825287 9663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d785596b4532417eacf100bdfcc4025d [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:41.825582 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:41.817319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40122 (local address 127.0.98.62:38961)
0926 11:30:41.817966 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:41.817982 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:41.818044 (+ 62us) server_negotiation.cc:407] Connection header received
0926 11:30:41.818846 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:41.818865 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:41.819197 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:41.819472 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:41.819881 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:41.820869 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:41.822109 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:41.822581 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:41.822701 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:41.823621 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:41.823625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:41.823638 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:30:41.824079 (+ 441us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:41.824134 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:30:41.824162 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:41.824238 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:41.824424 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 11:30:41.824690 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":169,"threads_started":1}
I20260926 11:30:41.826263 9663 raft_consensus.cc:504] T 00000000000000000000000000000000 P d785596b4532417eacf100bdfcc4025d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d785596b4532417eacf100bdfcc4025d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38961 } }
I20260926 11:30:41.826640 9663 leader_election.cc:302] T 00000000000000000000000000000000 P d785596b4532417eacf100bdfcc4025d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d785596b4532417eacf100bdfcc4025d; no voters:
I20260926 11:30:41.826992 9663 leader_election.cc:288] T 00000000000000000000000000000000 P d785596b4532417eacf100bdfcc4025d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:30:41.827097 9668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d785596b4532417eacf100bdfcc4025d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:30:41.827355 9668 raft_consensus.cc:686] T 00000000000000000000000000000000 P d785596b4532417eacf100bdfcc4025d [term 1 LEADER]: Becoming Leader. State: Replica: d785596b4532417eacf100bdfcc4025d, State: Running, Role: LEADER
I20260926 11:30:41.827692 9668 consensus_queue.cc:237] T 00000000000000000000000000000000 P d785596b4532417eacf100bdfcc4025d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d785596b4532417eacf100bdfcc4025d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38961 } }
I20260926 11:30:41.828263 9663 sys_catalog.cc:564] T 00000000000000000000000000000000 P d785596b4532417eacf100bdfcc4025d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:30:41.829474 9670 sys_catalog.cc:455] T 00000000000000000000000000000000 P d785596b4532417eacf100bdfcc4025d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d785596b4532417eacf100bdfcc4025d. Latest consensus state: current_term: 1 leader_uuid: "d785596b4532417eacf100bdfcc4025d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d785596b4532417eacf100bdfcc4025d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38961 } } }
I20260926 11:30:41.829455 9669 sys_catalog.cc:455] T 00000000000000000000000000000000 P d785596b4532417eacf100bdfcc4025d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d785596b4532417eacf100bdfcc4025d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d785596b4532417eacf100bdfcc4025d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38961 } } }
I20260926 11:30:41.829670 9669 sys_catalog.cc:458] T 00000000000000000000000000000000 P d785596b4532417eacf100bdfcc4025d [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:41.829620 9670 sys_catalog.cc:458] T 00000000000000000000000000000000 P d785596b4532417eacf100bdfcc4025d [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:41.830034 9677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:30:41.832962 9677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:30:41.838372 9677 catalog_manager.cc:1347] Generated new cluster ID: dbafc9be7d234c0fbb6a88a07c38401f
I20260926 11:30:41.838439 9677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:30:41.846017 9677 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:30:41.847196 9677 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:30:41.852015 9677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d785596b4532417eacf100bdfcc4025d: Generated new TSK 0
I20260926 11:30:41.852751 9677 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:30:41.855799 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38961
--builtin_ntp_servers=127.0.98.20:33843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 11:30:41.967123 9687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:41.967372 9687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:41.967443 9687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:41.971527 9687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:41.971603 9687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:41.971688 9687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:30:41.976471 9687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:41.977608 9687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:41.978988 9687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:41.988225 9695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:41.988258 9693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:41.988225 9692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:41.988519 9687 server_base.cc:1034] running on GCE node
I20260926 11:30:41.989032 9687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:41.989611 9687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:41.990778 9687 hybrid_clock.cc:648] HybridClock initialized: now 1790422241990760 us; error 34 us; skew 500 ppm
I20260926 11:30:41.992506 9687 webserver.cc:460] Webserver started at http://127.0.98.1:36775/ using document root <none> and password file <none>
I20260926 11:30:41.993016 9687 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:41.993081 9687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:41.993242 9687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:41.995153 9687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/data/instance:
uuid: "71cae8e9acf84cd7b65e59bdb29be9fd"
format_stamp: "Formatted at 2026-09-26 11:30:41 on dist-test-slave-mrck"
server_key: "6e0a6f8873a9c63f2ec97a4bda38e474"
server_key_iv: "23f70650e89ca7d82d378fb3e60db40f"
server_key_version: "encryptionkey@0"
I20260926 11:30:41.995647 9687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "71cae8e9acf84cd7b65e59bdb29be9fd"
format_stamp: "Formatted at 2026-09-26 11:30:41 on dist-test-slave-mrck"
server_key: "6e0a6f8873a9c63f2ec97a4bda38e474"
server_key_iv: "23f70650e89ca7d82d378fb3e60db40f"
server_key_version: "encryptionkey@0"
I20260926 11:30:41.999158 9687 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 11:30:42.001536 9701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:42.002697 9687 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:30:42.002789 9687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "71cae8e9acf84cd7b65e59bdb29be9fd"
format_stamp: "Formatted at 2026-09-26 11:30:41 on dist-test-slave-mrck"
server_key: "6e0a6f8873a9c63f2ec97a4bda38e474"
server_key_iv: "23f70650e89ca7d82d378fb3e60db40f"
server_key_version: "encryptionkey@0"
I20260926 11:30:42.002871 9687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:42.014978 9687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:42.015621 9687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:42.015856 9687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:42.016395 9687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:42.017243 9687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:42.017288 9687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:42.017325 9687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:42.017346 9687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:42.025529 9687 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46557
I20260926 11:30:42.025638 9814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46557 every 8 connection(s)
I20260926 11:30:42.026463 9687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:30:42.032199 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 9687
I20260926 11:30:42.032338 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:30:42.032665 392 external_mini_cluster.cc:1428] Setting key 442045a25983ec1504e35061f012ce5e
I20260926 11:30:42.034704 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38961
--builtin_ntp_servers=127.0.98.20:33843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 11:30:42.036360 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:42.028230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39801 (local address 127.0.98.62:38961)
0926 11:30:42.028359 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:42.028362 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:42.029566 (+ 1204us) server_negotiation.cc:407] Connection header received
0926 11:30:42.031026 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:42.031029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:42.031072 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:42.031146 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:42.032123 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:42.032871 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:42.034189 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:42.034417 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:42.034496 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:42.035722 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:42.035762 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:42.035769 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:30:42.035785 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:42.035811 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:30:42.035820 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:42.035879 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:42.036086 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 11:30:42.036175 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 11:30:42.037181 9817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:42.028682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38961 (local address 127.0.98.1:39801)
0926 11:30:42.029376 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:42.029414 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:42.030805 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:42.031299 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:42.031306 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:42.031406 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:42.031958 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:42.031978 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:42.033020 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:42.033023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:42.034058 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:42.034070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:42.034575 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:42.034580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:42.034783 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:42.035397 (+ 614us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:30:42.035402 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:42.035457 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:30:42.035539 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:42.035904 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:42.035915 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:42.035924 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 11:30:42.036072 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 11:30:42.036337 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":131,"threads_started":1}
I20260926 11:30:42.038385 9815 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38961
I20260926 11:30:42.038622 9815 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:42.039103 9815 heartbeater.cc:511] Master 127.0.98.62:38961 requested a full tablet report, sending...
I20260926 11:30:42.040756 9627 ts_manager.cc:194] Registered new tserver with Master: 71cae8e9acf84cd7b65e59bdb29be9fd (127.0.98.1:46557)
I20260926 11:30:42.042478 9627 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:39801
W20260926 11:30:42.144363 9818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:42.144603 9818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:42.144637 9818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:42.148542 9818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:42.148597 9818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:42.148656 9818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:30:42.153257 9818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:42.154204 9818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:42.155437 9818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:42.164531 9826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:42.164541 9823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:42.164511 9824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:42.164882 9818 server_base.cc:1034] running on GCE node
I20260926 11:30:42.165313 9818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:42.165897 9818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:42.167049 9818 hybrid_clock.cc:648] HybridClock initialized: now 1790422242167035 us; error 26 us; skew 500 ppm
I20260926 11:30:42.168983 9818 webserver.cc:460] Webserver started at http://127.0.98.2:33395/ using document root <none> and password file <none>
I20260926 11:30:42.169554 9818 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:42.169616 9818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:42.169842 9818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:42.171868 9818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/data/instance:
uuid: "48bd4b9f8507415087e8215389091a73"
format_stamp: "Formatted at 2026-09-26 11:30:42 on dist-test-slave-mrck"
server_key: "75ecb15d6e9d310b764dab2483ec0258"
server_key_iv: "8d447f4112a46a245a09323d512ec213"
server_key_version: "encryptionkey@0"
I20260926 11:30:42.172425 9818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "48bd4b9f8507415087e8215389091a73"
format_stamp: "Formatted at 2026-09-26 11:30:42 on dist-test-slave-mrck"
server_key: "75ecb15d6e9d310b764dab2483ec0258"
server_key_iv: "8d447f4112a46a245a09323d512ec213"
server_key_version: "encryptionkey@0"
I20260926 11:30:42.175949 9818 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:30:42.178411 9832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:42.179620 9818 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:30:42.179775 9818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "48bd4b9f8507415087e8215389091a73"
format_stamp: "Formatted at 2026-09-26 11:30:42 on dist-test-slave-mrck"
server_key: "75ecb15d6e9d310b764dab2483ec0258"
server_key_iv: "8d447f4112a46a245a09323d512ec213"
server_key_version: "encryptionkey@0"
I20260926 11:30:42.179873 9818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:42.198341 9818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:42.199072 9818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:42.199271 9818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:42.200074 9818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:42.201323 9818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:42.201392 9818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:42.201463 9818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:42.201499 9818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:42.210783 9818 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44565
I20260926 11:30:42.210839 9945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44565 every 8 connection(s)
I20260926 11:30:42.211691 9818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:30:42.220577 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:42.213430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42747 (local address 127.0.98.62:38961)
0926 11:30:42.213555 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:42.213558 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:42.214535 (+ 977us) server_negotiation.cc:407] Connection header received
0926 11:30:42.215949 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:42.215953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:42.216011 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:42.216089 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:42.216938 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:42.217471 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:42.218615 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:42.218852 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:42.218964 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:42.220094 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:42.220097 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:42.220100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:30:42.220113 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:42.220140 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:30:42.220148 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:42.220227 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:42.220376 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 11:30:42.220436 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 11:30:42.221107 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 9818
I20260926 11:30:42.221243 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:30:42.221451 9948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:42.213773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38961 (local address 127.0.98.2:42747)
0926 11:30:42.214392 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:42.214431 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:42.215693 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:42.216253 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:42.216260 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:42.216339 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:42.216797 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:42.216808 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:42.217591 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:42.217594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:42.218490 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:42.218498 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:42.218998 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:42.219001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:42.219171 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:42.219839 (+ 668us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:30:42.219840 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:42.219878 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:30:42.219936 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:42.220228 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:42.220238 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:42.220246 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 11:30:42.220376 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 11:30:42.220638 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":110,"threads_started":1}
I20260926 11:30:42.221554 392 external_mini_cluster.cc:1428] Setting key 5fc69b7744b71b215c67810ea9c62872
I20260926 11:30:42.222628 9946 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38961
I20260926 11:30:42.223249 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38961
--builtin_ntp_servers=127.0.98.20:33843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 11:30:42.223914 9946 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:42.224414 9946 heartbeater.cc:511] Master 127.0.98.62:38961 requested a full tablet report, sending...
I20260926 11:30:42.225446 9627 ts_manager.cc:194] Registered new tserver with Master: 48bd4b9f8507415087e8215389091a73 (127.0.98.2:44565)
I20260926 11:30:42.226243 9627 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:42747
W20260926 11:30:42.333696 9949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:42.333952 9949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:42.333988 9949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:42.337893 9949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:42.337949 9949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:42.338001 9949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:30:42.342607 9949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:42.343626 9949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:42.345479 9949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:42.355225 9957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:42.355217 9954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:42.355217 9955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:42.355324 9949 server_base.cc:1034] running on GCE node
I20260926 11:30:42.355943 9949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:42.356628 9949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:42.357828 9949 hybrid_clock.cc:648] HybridClock initialized: now 1790422242357813 us; error 30 us; skew 500 ppm
I20260926 11:30:42.359896 9949 webserver.cc:460] Webserver started at http://127.0.98.3:37599/ using document root <none> and password file <none>
I20260926 11:30:42.360499 9949 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:42.360597 9949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:42.360842 9949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:42.362941 9949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/data/instance:
uuid: "38eef1cb0dba4f1485a6e915f995376f"
format_stamp: "Formatted at 2026-09-26 11:30:42 on dist-test-slave-mrck"
server_key: "1c1dab49f388b9ad8bd8204104bdd439"
server_key_iv: "0c84ea668d61037bfdbe5b97c655f980"
server_key_version: "encryptionkey@0"
I20260926 11:30:42.363518 9949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "38eef1cb0dba4f1485a6e915f995376f"
format_stamp: "Formatted at 2026-09-26 11:30:42 on dist-test-slave-mrck"
server_key: "1c1dab49f388b9ad8bd8204104bdd439"
server_key_iv: "0c84ea668d61037bfdbe5b97c655f980"
server_key_version: "encryptionkey@0"
I20260926 11:30:42.367280 9949 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260926 11:30:42.370013 9963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:42.371300 9949 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:30:42.371447 9949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "38eef1cb0dba4f1485a6e915f995376f"
format_stamp: "Formatted at 2026-09-26 11:30:42 on dist-test-slave-mrck"
server_key: "1c1dab49f388b9ad8bd8204104bdd439"
server_key_iv: "0c84ea668d61037bfdbe5b97c655f980"
server_key_version: "encryptionkey@0"
I20260926 11:30:42.371570 9949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:42.399580 9949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:42.400382 9949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:42.400586 9949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:42.401188 9949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:42.402160 9949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:42.402231 9949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:42.402307 9949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:42.402340 9949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:42.411116 9949 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42901
I20260926 11:30:42.411139 10076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42901 every 8 connection(s)
I20260926 11:30:42.412083 9949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:30:42.420179 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 9949
I20260926 11:30:42.420337 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:30:42.420679 392 external_mini_cluster.cc:1428] Setting key 36378163d9a29387a1f20a6b2e97fe13
I20260926 11:30:42.421427 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:42.413957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55783 (local address 127.0.98.62:38961)
0926 11:30:42.414112 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:42.414115 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:42.415163 (+ 1048us) server_negotiation.cc:407] Connection header received
0926 11:30:42.416614 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:42.416618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:42.416681 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:42.416789 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:42.417669 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:42.418202 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:42.419331 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:42.419555 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:42.419675 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:42.420917 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:42.420920 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:42.420922 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:30:42.420934 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:42.420952 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:30:42.420961 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:42.421066 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:42.421223 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 11:30:42.421288 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 11:30:42.422354 10079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:42.414382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38961 (local address 127.0.98.3:55783)
0926 11:30:42.415011 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:42.415048 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:42.416402 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:42.416953 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:42.416961 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:42.417047 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:42.417517 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:42.417529 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:42.418333 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:42.418339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:42.419209 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:42.419217 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:42.419702 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:42.419706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:42.419894 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:42.420626 (+ 732us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:30:42.420629 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:42.420682 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:30:42.420765 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:42.421072 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:42.421079 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:42.421083 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 11:30:42.421220 (+ 137us) client_negotiation.cc:240] Negotiation successful
0926 11:30:42.421491 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":103,"threads_started":1}
I20260926 11:30:42.423396 10077 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38961
I20260926 11:30:42.423633 10077 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:42.424155 10077 heartbeater.cc:511] Master 127.0.98.62:38961 requested a full tablet report, sending...
I20260926 11:30:42.425125 9627 ts_manager.cc:194] Registered new tserver with Master: 38eef1cb0dba4f1485a6e915f995376f (127.0.98.3:42901)
I20260926 11:30:42.425698 9627 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:55783
I20260926 11:30:42.434710 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 11:30:42.437561 9666 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:42.436675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40126 (local address 127.0.98.62:38961)
0926 11:30:42.436813 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:42.436818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:42.436993 (+ 175us) server_negotiation.cc:407] Connection header received
0926 11:30:42.437081 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:42.437083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:42.437123 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:42.437195 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:42.437416 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40126: BlockingRecv error: recv got EOF from 127.0.0.1:40126 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 11:30:42.437628 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 9687
I20260926 11:30:42.442993 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 9818
I20260926 11:30:42.448256 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 9949
I20260926 11:30:42.453377 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 9596
2026-09-26T11:30:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (841 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T11:30:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:30:42Z Disabled control of system clock
I20260926 11:30:42.482508 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39233
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 11:30:42.592657 10088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:42.592926 10088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:42.592967 10088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:42.596827 10088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:30:42.596884 10088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:42.596901 10088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:42.596921 10088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:30:42.596939 10088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:30:42.601688 10088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39233
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:42.602810 10088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:42.604190 10088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:42.612429 10094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:42.612515 10093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:42.612669 10096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:42.612859 10088 server_base.cc:1034] running on GCE node
I20260926 11:30:42.613517 10088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:42.614579 10088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:42.615792 10088 hybrid_clock.cc:648] HybridClock initialized: now 1790422242615754 us; error 45 us; skew 500 ppm
I20260926 11:30:42.617619 10088 webserver.cc:460] Webserver started at http://127.0.98.62:45267/ using document root <none> and password file <none>
I20260926 11:30:42.618134 10088 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:42.618191 10088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:42.618343 10088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:42.620306 10088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "a136ac41d0f04f3bb65325eb86999448"
format_stamp: "Formatted at 2026-09-26 11:30:42 on dist-test-slave-mrck"
server_key: "104989fc5a492fe8639f8182d14144c5"
server_key_iv: "b7e1b22b0438e1dfeaefb66058ce594e"
server_key_version: "encryptionkey@0"
I20260926 11:30:42.620790 10088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "a136ac41d0f04f3bb65325eb86999448"
format_stamp: "Formatted at 2026-09-26 11:30:42 on dist-test-slave-mrck"
server_key: "104989fc5a492fe8639f8182d14144c5"
server_key_iv: "b7e1b22b0438e1dfeaefb66058ce594e"
server_key_version: "encryptionkey@0"
I20260926 11:30:42.624300 10088 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:30:42.626557 10102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:42.628013 10088 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:30:42.628136 10088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "a136ac41d0f04f3bb65325eb86999448"
format_stamp: "Formatted at 2026-09-26 11:30:42 on dist-test-slave-mrck"
server_key: "104989fc5a492fe8639f8182d14144c5"
server_key_iv: "b7e1b22b0438e1dfeaefb66058ce594e"
server_key_version: "encryptionkey@0"
I20260926 11:30:42.628229 10088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:42.632679 10088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:42.633239 10088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:42.633373 10088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:42.640187 10088 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39233
I20260926 11:30:42.640322 10154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39233 every 8 connection(s)
I20260926 11:30:42.641153 10088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:30:42.644361 10155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:42.648181 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 10088
I20260926 11:30:42.648298 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:30:42.648617 392 external_mini_cluster.cc:1428] Setting key 3a63a3d6706305c249b5aba8fb6b6eef
I20260926 11:30:42.650390 10155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a136ac41d0f04f3bb65325eb86999448: Bootstrap starting.
I20260926 11:30:42.652812 10155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a136ac41d0f04f3bb65325eb86999448: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:42.653714 10155 log.cc:824] T 00000000000000000000000000000000 P a136ac41d0f04f3bb65325eb86999448: Log is configured to *not* fsync() on all Append() calls
I20260926 11:30:42.654487 10158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:42.650001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57816 (local address 127.0.98.62:39233)
0926 11:30:42.650701 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:42.650717 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:42.650753 (+ 36us) server_negotiation.cc:407] Connection header received
0926 11:30:42.651395 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:42.651411 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:42.651832 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:42.652205 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:42.653010 (+ 805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:42.653017 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:42.653035 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:30:42.653049 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:42.653124 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:30:42.653157 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:42.653272 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:42.653370 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 11:30:42.653615 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":145,"threads_started":1}
I20260926 11:30:42.655905 10155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a136ac41d0f04f3bb65325eb86999448: No bootstrap required, opened a new log
I20260926 11:30:42.659296 10155 raft_consensus.cc:348] T 00000000000000000000000000000000 P a136ac41d0f04f3bb65325eb86999448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a136ac41d0f04f3bb65325eb86999448" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39233 } }
I20260926 11:30:42.659499 10155 raft_consensus.cc:374] T 00000000000000000000000000000000 P a136ac41d0f04f3bb65325eb86999448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:42.659569 10155 raft_consensus.cc:729] T 00000000000000000000000000000000 P a136ac41d0f04f3bb65325eb86999448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a136ac41d0f04f3bb65325eb86999448, State: Initialized, Role: FOLLOWER
I20260926 11:30:42.660046 10155 consensus_queue.cc:260] T 00000000000000000000000000000000 P a136ac41d0f04f3bb65325eb86999448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a136ac41d0f04f3bb65325eb86999448" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39233 } }
I20260926 11:30:42.660167 10155 raft_consensus.cc:388] T 00000000000000000000000000000000 P a136ac41d0f04f3bb65325eb86999448 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:30:42.660248 10155 raft_consensus.cc:482] T 00000000000000000000000000000000 P a136ac41d0f04f3bb65325eb86999448 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:30:42.660338 10155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a136ac41d0f04f3bb65325eb86999448 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:42.661257 10155 raft_consensus.cc:504] T 00000000000000000000000000000000 P a136ac41d0f04f3bb65325eb86999448 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a136ac41d0f04f3bb65325eb86999448" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39233 } }
I20260926 11:30:42.661650 10155 leader_election.cc:302] T 00000000000000000000000000000000 P a136ac41d0f04f3bb65325eb86999448 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a136ac41d0f04f3bb65325eb86999448; no voters:
I20260926 11:30:42.661984 10155 leader_election.cc:288] T 00000000000000000000000000000000 P a136ac41d0f04f3bb65325eb86999448 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:30:42.662153 10160 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a136ac41d0f04f3bb65325eb86999448 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:30:42.662384 10160 raft_consensus.cc:686] T 00000000000000000000000000000000 P a136ac41d0f04f3bb65325eb86999448 [term 1 LEADER]: Becoming Leader. State: Replica: a136ac41d0f04f3bb65325eb86999448, State: Running, Role: LEADER
I20260926 11:30:42.662662 10160 consensus_queue.cc:237] T 00000000000000000000000000000000 P a136ac41d0f04f3bb65325eb86999448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a136ac41d0f04f3bb65325eb86999448" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39233 } }
I20260926 11:30:42.663210 10155 sys_catalog.cc:564] T 00000000000000000000000000000000 P a136ac41d0f04f3bb65325eb86999448 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:30:42.664304 10161 sys_catalog.cc:455] T 00000000000000000000000000000000 P a136ac41d0f04f3bb65325eb86999448 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a136ac41d0f04f3bb65325eb86999448" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a136ac41d0f04f3bb65325eb86999448" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39233 } } }
I20260926 11:30:42.664430 10161 sys_catalog.cc:458] T 00000000000000000000000000000000 P a136ac41d0f04f3bb65325eb86999448 [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:42.664635 10162 sys_catalog.cc:455] T 00000000000000000000000000000000 P a136ac41d0f04f3bb65325eb86999448 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a136ac41d0f04f3bb65325eb86999448. Latest consensus state: current_term: 1 leader_uuid: "a136ac41d0f04f3bb65325eb86999448" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a136ac41d0f04f3bb65325eb86999448" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39233 } } }
I20260926 11:30:42.664786 10162 sys_catalog.cc:458] T 00000000000000000000000000000000 P a136ac41d0f04f3bb65325eb86999448 [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:42.664810 10169 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:30:42.668376 10169 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:30:42.673776 10169 catalog_manager.cc:1347] Generated new cluster ID: e68cd58b773e450b89f36b149a6b2c59
I20260926 11:30:42.673846 10169 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:30:42.695864 10169 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:30:42.696223 10169 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:30:42.709777 10169 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a136ac41d0f04f3bb65325eb86999448: Generated new TSK 0
I20260926 11:30:42.710636 10169 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:30:42.723177 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39233
--builtin_ntp_servers=127.0.98.20:40755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 11:30:42.834115 10179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:42.834360 10179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:42.834400 10179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:42.838493 10179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:42.838541 10179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:42.838600 10179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:30:42.843294 10179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10179
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:42.844362 10179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:42.845640 10179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:42.855062 10184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:42.855063 10187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:42.855094 10179 server_base.cc:1034] running on GCE node
W20260926 11:30:42.855103 10185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:42.855834 10179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:42.856402 10179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:42.857585 10179 hybrid_clock.cc:648] HybridClock initialized: now 1790422242857549 us; error 46 us; skew 500 ppm
I20260926 11:30:42.859488 10179 webserver.cc:460] Webserver started at http://127.0.98.1:33269/ using document root <none> and password file <none>
I20260926 11:30:42.860141 10179 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:42.860204 10179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:42.860419 10179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:42.862476 10179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/data/instance:
uuid: "f85bdba9cea84527be5a3536710edd49"
format_stamp: "Formatted at 2026-09-26 11:30:42 on dist-test-slave-mrck"
server_key: "aeb43acd49d5cb0f49eea1da06eb681a"
server_key_iv: "b8d454f35f3f54c18a417956ec48c953"
server_key_version: "encryptionkey@0"
I20260926 11:30:42.863042 10179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f85bdba9cea84527be5a3536710edd49"
format_stamp: "Formatted at 2026-09-26 11:30:42 on dist-test-slave-mrck"
server_key: "aeb43acd49d5cb0f49eea1da06eb681a"
server_key_iv: "b8d454f35f3f54c18a417956ec48c953"
server_key_version: "encryptionkey@0"
I20260926 11:30:42.866823 10179 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 11:30:42.869320 10193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:42.870527 10179 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 11:30:42.870661 10179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "f85bdba9cea84527be5a3536710edd49"
format_stamp: "Formatted at 2026-09-26 11:30:42 on dist-test-slave-mrck"
server_key: "aeb43acd49d5cb0f49eea1da06eb681a"
server_key_iv: "b8d454f35f3f54c18a417956ec48c953"
server_key_version: "encryptionkey@0"
I20260926 11:30:42.870774 10179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:42.875350 10179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:42.875959 10179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:42.876178 10179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:42.876704 10179 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:42.877674 10179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:42.877740 10179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:42.877805 10179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:42.877851 10179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:42.886951 10179 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33871
I20260926 11:30:42.886973 10306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33871 every 8 connection(s)
I20260926 11:30:42.887938 10179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:30:42.889060 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 10179
I20260926 11:30:42.889155 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:30:42.889499 392 external_mini_cluster.cc:1428] Setting key 849e10e763ffe12563c48bf02cc14230
I20260926 11:30:42.891693 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39233
--builtin_ntp_servers=127.0.98.20:40755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 11:30:42.894904 10158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:42.889813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53643 (local address 127.0.98.62:39233)
0926 11:30:42.890176 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:42.890181 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:42.891263 (+ 1082us) server_negotiation.cc:407] Connection header received
0926 11:30:42.893141 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:42.893146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:42.893196 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:42.893303 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:42.894395 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:42.894398 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:42.894400 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:30:42.894410 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:42.894427 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:30:42.894434 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:42.894543 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:42.894762 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 11:30:42.894803 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256}
I20260926 11:30:42.896057 10309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:42.890297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39233 (local address 127.0.98.1:53643)
0926 11:30:42.891089 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:42.891152 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:42.892900 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:42.893455 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:42.893461 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:42.893550 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:42.894151 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:30:42.894153 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:42.894193 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:30:42.894261 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:42.894563 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:42.894573 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:42.894582 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 11:30:42.894766 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 11:30:42.895004 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":134,"threads_started":1}
I20260926 11:30:42.897321 10307 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39233
I20260926 11:30:42.897576 10307 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:42.898079 10307 heartbeater.cc:511] Master 127.0.98.62:39233 requested a full tablet report, sending...
I20260926 11:30:42.899847 10119 ts_manager.cc:194] Registered new tserver with Master: f85bdba9cea84527be5a3536710edd49 (127.0.98.1:33871)
W20260926 11:30:43.002455 10310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:43.002737 10310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:43.002777 10310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:43.006742 10310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:43.006808 10310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:43.006865 10310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:30:43.011601 10310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:43.012718 10310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:43.014020 10310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:43.023530 10316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:43.023554 10315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:43.023753 10310 server_base.cc:1034] running on GCE node
W20260926 11:30:43.023532 10318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:43.024307 10310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:43.024900 10310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:43.026079 10310 hybrid_clock.cc:648] HybridClock initialized: now 1790422243026045 us; error 45 us; skew 500 ppm
I20260926 11:30:43.027976 10310 webserver.cc:460] Webserver started at http://127.0.98.2:41245/ using document root <none> and password file <none>
I20260926 11:30:43.028503 10310 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:43.028561 10310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:43.028723 10310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:43.030663 10310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/data/instance:
uuid: "1f99c6fd7b9341f19fb72fb5ad3a81a1"
format_stamp: "Formatted at 2026-09-26 11:30:43 on dist-test-slave-mrck"
server_key: "ef3f12a29a1ce4366b14200619bf18d7"
server_key_iv: "67cfb10cf9797fcb246554a6658af1d4"
server_key_version: "encryptionkey@0"
I20260926 11:30:43.031169 10310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1f99c6fd7b9341f19fb72fb5ad3a81a1"
format_stamp: "Formatted at 2026-09-26 11:30:43 on dist-test-slave-mrck"
server_key: "ef3f12a29a1ce4366b14200619bf18d7"
server_key_iv: "67cfb10cf9797fcb246554a6658af1d4"
server_key_version: "encryptionkey@0"
I20260926 11:30:43.034773 10310 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 11:30:43.037087 10324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:43.038365 10310 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 11:30:43.038461 10310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "1f99c6fd7b9341f19fb72fb5ad3a81a1"
format_stamp: "Formatted at 2026-09-26 11:30:43 on dist-test-slave-mrck"
server_key: "ef3f12a29a1ce4366b14200619bf18d7"
server_key_iv: "67cfb10cf9797fcb246554a6658af1d4"
server_key_version: "encryptionkey@0"
I20260926 11:30:43.038547 10310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:43.043022 10310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:43.043577 10310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:43.043715 10310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:43.044272 10310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:43.045109 10310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:43.045163 10310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:43.045200 10310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:43.045217 10310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:43.053716 10310 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46015
I20260926 11:30:43.053797 10437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46015 every 8 connection(s)
I20260926 11:30:43.054632 10310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:30:43.058526 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 10310
I20260926 11:30:43.058663 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:30:43.058943 392 external_mini_cluster.cc:1428] Setting key c5153888b036ce1c413e0a2c339532fd
I20260926 11:30:43.060869 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39233
--builtin_ntp_servers=127.0.98.20:40755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 11:30:43.061127 10158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:43.056529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37397 (local address 127.0.98.62:39233)
0926 11:30:43.056691 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:43.056701 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:43.057689 (+ 988us) server_negotiation.cc:407] Connection header received
0926 11:30:43.059192 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:43.059195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:43.059246 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:43.059360 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:43.060616 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:43.060621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:43.060624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:30:43.060643 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:43.060663 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:30:43.060673 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:43.060772 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:43.060923 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 11:30:43.060982 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 11:30:43.061986 10440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:43.056883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39233 (local address 127.0.98.2:37397)
0926 11:30:43.057528 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:43.057570 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:43.058937 (+ 1367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:43.059534 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:43.059542 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:43.059631 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:43.060353 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:30:43.060355 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:43.060396 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:30:43.060465 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:43.060760 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:43.060769 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:43.060775 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 11:30:43.060982 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 11:30:43.061247 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":100,"threads_started":1}
I20260926 11:30:43.063235 10438 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39233
I20260926 11:30:43.063464 10438 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:43.063921 10438 heartbeater.cc:511] Master 127.0.98.62:39233 requested a full tablet report, sending...
I20260926 11:30:43.064929 10119 ts_manager.cc:194] Registered new tserver with Master: 1f99c6fd7b9341f19fb72fb5ad3a81a1 (127.0.98.2:46015)
W20260926 11:30:43.169960 10441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:43.170238 10441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:43.170315 10441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:43.174460 10441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:43.174543 10441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:43.174624 10441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:30:43.179554 10441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:43.180639 10441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:43.182057 10441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:43.191325 10446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:43.191332 10447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:43.191308 10449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:43.191493 10441 server_base.cc:1034] running on GCE node
I20260926 11:30:43.192003 10441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:43.192835 10441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:43.194044 10441 hybrid_clock.cc:648] HybridClock initialized: now 1790422243194002 us; error 51 us; skew 500 ppm
I20260926 11:30:43.195873 10441 webserver.cc:460] Webserver started at http://127.0.98.3:35455/ using document root <none> and password file <none>
I20260926 11:30:43.196442 10441 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:43.196504 10441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:43.196712 10441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:43.198702 10441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/data/instance:
uuid: "3c4bce9e67114c31bfaaafc63d048a42"
format_stamp: "Formatted at 2026-09-26 11:30:43 on dist-test-slave-mrck"
server_key: "9e2d8f3b14add9c01b8ad552c148e5fc"
server_key_iv: "3ec7306c37730770a213b5909d83fd39"
server_key_version: "encryptionkey@0"
I20260926 11:30:43.199259 10441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3c4bce9e67114c31bfaaafc63d048a42"
format_stamp: "Formatted at 2026-09-26 11:30:43 on dist-test-slave-mrck"
server_key: "9e2d8f3b14add9c01b8ad552c148e5fc"
server_key_iv: "3ec7306c37730770a213b5909d83fd39"
server_key_version: "encryptionkey@0"
I20260926 11:30:43.202869 10441 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260926 11:30:43.206660 10455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:43.208081 10441 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260926 11:30:43.208175 10441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "3c4bce9e67114c31bfaaafc63d048a42"
format_stamp: "Formatted at 2026-09-26 11:30:43 on dist-test-slave-mrck"
server_key: "9e2d8f3b14add9c01b8ad552c148e5fc"
server_key_iv: "3ec7306c37730770a213b5909d83fd39"
server_key_version: "encryptionkey@0"
I20260926 11:30:43.208268 10441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:43.212849 10441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:43.213366 10441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:43.213490 10441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:43.213986 10441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:43.214793 10441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:43.214838 10441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:43.214874 10441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:43.214895 10441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:43.223485 10441 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36567
I20260926 11:30:43.223553 10568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36567 every 8 connection(s)
I20260926 11:30:43.224438 10441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:30:43.227131 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 10441
I20260926 11:30:43.227242 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790422180210734-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:30:43.227586 392 external_mini_cluster.cc:1428] Setting key b407a5113e87f3ea31a0ff78eb62cfd6
I20260926 11:30:43.230666 10158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:43.226206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42999 (local address 127.0.98.62:39233)
0926 11:30:43.226349 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:43.226353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:43.227512 (+ 1159us) server_negotiation.cc:407] Connection header received
0926 11:30:43.229003 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:43.229007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:43.229057 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:43.229141 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:43.230218 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:43.230222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:43.230225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:30:43.230236 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:43.230255 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:30:43.230264 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:43.230361 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:43.230499 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 11:30:43.230542 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 11:30:43.231594 10571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:43.226588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39233 (local address 127.0.98.3:42999)
0926 11:30:43.227336 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:43.227393 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:43.228792 (+ 1399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:43.229312 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:43.229318 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:43.229404 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:43.229975 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:30:43.229977 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:43.230018 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:30:43.230083 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:43.230350 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:43.230358 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:43.230363 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 11:30:43.230552 (+ 189us) client_negotiation.cc:240] Negotiation successful
0926 11:30:43.230785 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":95,"threads_started":1}
I20260926 11:30:43.232585 10569 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39233
I20260926 11:30:43.232815 10569 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:43.233196 10569 heartbeater.cc:511] Master 127.0.98.62:39233 requested a full tablet report, sending...
I20260926 11:30:43.234127 10119 ts_manager.cc:194] Registered new tserver with Master: 3c4bce9e67114c31bfaaafc63d048a42 (127.0.98.3:36567)
I20260926 11:30:43.241375 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 11:30:43.244372 10158 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:30:43.243331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57832 (local address 127.0.98.62:39233)
0926 11:30:43.243481 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:43.243486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:43.243672 (+ 186us) server_negotiation.cc:407] Connection header received
0926 11:30:43.243882 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:43.243886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:43.243928 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:43.244013 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:43.244249 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57832: BlockingRecv error: recv got EOF from 127.0.0.1:57832 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 11:30:43.244481 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 10179
I20260926 11:30:43.249974 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 10310
I20260926 11:30:43.255232 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 10441
I20260926 11:30:43.260526 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 10088
2026-09-26T11:30:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (807 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:30:43 dist-test-slave-mrck krb5kdc[10579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:30:43 dist-test-slave-mrck krb5kdc[10579](info): set up 2 sockets
Sep 26 11:30:43 dist-test-slave-mrck krb5kdc[10579](info): commencing operation
krb5kdc: starting...
W20260926 11:30:44.288744 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:30:44 dist-test-slave-mrck krb5kdc[10579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:30:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:30:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:30:44.406035 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40741
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40741
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:44.513067 10592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:44.513319 10592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:44.513357 10592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:44.517333 10592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:30:44.517383 10592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:44.517402 10592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:44.517418 10592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:30:44.517433 10592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:30:44.522694 10592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:44.523685 10592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:44.524713 10592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:44.532900 10600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:44.532910 10598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:44.533039 10592 server_base.cc:1034] running on GCE node
W20260926 11:30:44.532913 10597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:44.533739 10592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:44.534798 10592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:44.535974 10592 hybrid_clock.cc:648] HybridClock initialized: now 1790422244535961 us; error 25 us; skew 500 ppm
Sep 26 11:30:44 dist-test-slave-mrck krb5kdc[10579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:44.545166 10592 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:30:44.546031 10592 webserver.cc:460] Webserver started at http://127.0.98.62:42915/ using document root <none> and password file <none>
I20260926 11:30:44.546541 10592 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:44.546597 10592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:44.546767 10592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:44.548820 10592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "a0be1489984040438a61f0f4e07751c7"
format_stamp: "Formatted at 2026-09-26 11:30:44 on dist-test-slave-mrck"
server_key: "e532cfcfb8286cf61f35642079d5ec08"
server_key_iv: "0a777f3c481be4253d656a36b853540f"
server_key_version: "encryptionkey@0"
I20260926 11:30:44.549312 10592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "a0be1489984040438a61f0f4e07751c7"
format_stamp: "Formatted at 2026-09-26 11:30:44 on dist-test-slave-mrck"
server_key: "e532cfcfb8286cf61f35642079d5ec08"
server_key_iv: "0a777f3c481be4253d656a36b853540f"
server_key_version: "encryptionkey@0"
I20260926 11:30:44.552790 10592 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260926 11:30:44.554997 10607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:44.556195 10592 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 11:30:44.556315 10592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "a0be1489984040438a61f0f4e07751c7"
format_stamp: "Formatted at 2026-09-26 11:30:44 on dist-test-slave-mrck"
server_key: "e532cfcfb8286cf61f35642079d5ec08"
server_key_iv: "0a777f3c481be4253d656a36b853540f"
server_key_version: "encryptionkey@0"
I20260926 11:30:44.556409 10592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:44.566690 10592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:44.569954 10592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:44.570163 10592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:44.576716 10592 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40741
I20260926 11:30:44.577005 10659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40741 every 8 connection(s)
I20260926 11:30:44.577690 10592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:30:44.580407 10660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:44.582007 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 10592
I20260926 11:30:44.582130 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:30:44.582413 392 external_mini_cluster.cc:1428] Setting key cf18e5e5920246dc351f4e0a53ffc622
I20260926 11:30:44.586694 10660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0be1489984040438a61f0f4e07751c7: Bootstrap starting.
I20260926 11:30:44.589159 10660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0be1489984040438a61f0f4e07751c7: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:44.590042 10660 log.cc:824] T 00000000000000000000000000000000 P a0be1489984040438a61f0f4e07751c7: Log is configured to *not* fsync() on all Append() calls
Sep 26 11:30:44 dist-test-slave-mrck krb5kdc[10579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:30:44.592082 10660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0be1489984040438a61f0f4e07751c7: No bootstrap required, opened a new log
I20260926 11:30:44.596706 10663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:44.583692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48430 (local address 127.0.98.62:40741)
0926 11:30:44.584156 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:44.584171 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:44.584204 (+ 33us) server_negotiation.cc:407] Connection header received
0926 11:30:44.584797 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:44.584811 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:44.585109 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:44.585409 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:44.586283 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:44.587272 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:44.588432 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:44.588865 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:44.588976 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:44.591633 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:44.591640 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:44.591656 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:44.591684 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:44.593688 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:44.594098 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:44.594104 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:44.594110 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:44.594271 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:44.594569 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:44.594575 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:44.594576 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:44.595067 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:44.595260 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:44.595440 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 11:30:44.595722 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":130,"threads_started":1}
I20260926 11:30:44.596791 10660 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0be1489984040438a61f0f4e07751c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0be1489984040438a61f0f4e07751c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40741 } }
I20260926 11:30:44.597038 10660 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0be1489984040438a61f0f4e07751c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:44.597090 10660 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0be1489984040438a61f0f4e07751c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0be1489984040438a61f0f4e07751c7, State: Initialized, Role: FOLLOWER
I20260926 11:30:44.597509 10660 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0be1489984040438a61f0f4e07751c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0be1489984040438a61f0f4e07751c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40741 } }
I20260926 11:30:44.597668 10660 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0be1489984040438a61f0f4e07751c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:30:44.597735 10660 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0be1489984040438a61f0f4e07751c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:30:44.597824 10660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0be1489984040438a61f0f4e07751c7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:44.598659 10660 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0be1489984040438a61f0f4e07751c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0be1489984040438a61f0f4e07751c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40741 } }
I20260926 11:30:44.598981 10660 leader_election.cc:302] T 00000000000000000000000000000000 P a0be1489984040438a61f0f4e07751c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0be1489984040438a61f0f4e07751c7; no voters:
I20260926 11:30:44.599231 10660 leader_election.cc:288] T 00000000000000000000000000000000 P a0be1489984040438a61f0f4e07751c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:30:44.599370 10665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0be1489984040438a61f0f4e07751c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:30:44.599606 10665 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0be1489984040438a61f0f4e07751c7 [term 1 LEADER]: Becoming Leader. State: Replica: a0be1489984040438a61f0f4e07751c7, State: Running, Role: LEADER
I20260926 11:30:44.599929 10665 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0be1489984040438a61f0f4e07751c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0be1489984040438a61f0f4e07751c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40741 } }
I20260926 11:30:44.600399 10660 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0be1489984040438a61f0f4e07751c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:30:44.601486 10667 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0be1489984040438a61f0f4e07751c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0be1489984040438a61f0f4e07751c7. Latest consensus state: current_term: 1 leader_uuid: "a0be1489984040438a61f0f4e07751c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0be1489984040438a61f0f4e07751c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40741 } } }
I20260926 11:30:44.601604 10667 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0be1489984040438a61f0f4e07751c7 [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:44.601876 10666 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0be1489984040438a61f0f4e07751c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0be1489984040438a61f0f4e07751c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0be1489984040438a61f0f4e07751c7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40741 } } }
I20260926 11:30:44.601963 10666 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0be1489984040438a61f0f4e07751c7 [sys.catalog]: This master's current role is: LEADER
I20260926 11:30:44.602078 10674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:30:44.605065 10674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:30:44.609894 10674 catalog_manager.cc:1347] Generated new cluster ID: 2a2cf923204347e186a7e0d7d3148496
I20260926 11:30:44.609956 10674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:30:44.621994 10674 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:30:44.622967 10674 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:30:44.630959 10674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0be1489984040438a61f0f4e07751c7: Generated new TSK 0
I20260926 11:30:44.631598 10674 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:30:44.662801 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40741
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:44.776789 10686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:44.777031 10686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:44.777072 10686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:44.781270 10686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:44.781327 10686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:44.781418 10686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:30:44.786711 10686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:44.787717 10686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:44.788796 10686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:44.797907 10692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:44.797912 10694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:44.798099 10686 server_base.cc:1034] running on GCE node
W20260926 11:30:44.798291 10691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:44.798638 10686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:44.799268 10686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:44.800421 10686 hybrid_clock.cc:648] HybridClock initialized: now 1790422244800407 us; error 27 us; skew 500 ppm
Sep 26 11:30:44 dist-test-slave-mrck krb5kdc[10579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:44.803493 10686 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:30:44.804355 10686 webserver.cc:460] Webserver started at http://127.0.98.1:44651/ using document root <none> and password file <none>
I20260926 11:30:44.804857 10686 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:44.804912 10686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:44.805065 10686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:44.806977 10686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-0/data/instance:
uuid: "cdb64b98fa5741e7a728e51422f4c3ff"
format_stamp: "Formatted at 2026-09-26 11:30:44 on dist-test-slave-mrck"
server_key: "880e1f1ea83673e3c3ef4fd13e8b4a2f"
server_key_iv: "8a354755189f16c437325086691b3d4a"
server_key_version: "encryptionkey@0"
I20260926 11:30:44.807456 10686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cdb64b98fa5741e7a728e51422f4c3ff"
format_stamp: "Formatted at 2026-09-26 11:30:44 on dist-test-slave-mrck"
server_key: "880e1f1ea83673e3c3ef4fd13e8b4a2f"
server_key_iv: "8a354755189f16c437325086691b3d4a"
server_key_version: "encryptionkey@0"
I20260926 11:30:44.810937 10686 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 11:30:44.813225 10701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:44.814420 10686 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 11:30:44.814536 10686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "cdb64b98fa5741e7a728e51422f4c3ff"
format_stamp: "Formatted at 2026-09-26 11:30:44 on dist-test-slave-mrck"
server_key: "880e1f1ea83673e3c3ef4fd13e8b4a2f"
server_key_iv: "8a354755189f16c437325086691b3d4a"
server_key_version: "encryptionkey@0"
I20260926 11:30:44.814647 10686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:44.828693 10686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:44.831857 10686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:44.832049 10686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:44.832615 10686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:44.833405 10686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:44.833451 10686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:44.833513 10686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:44.833542 10686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:44.841856 10686 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41259
I20260926 11:30:44.841944 10814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41259 every 8 connection(s)
I20260926 11:30:44.842725 10686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:30:44.849068 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 10686
I20260926 11:30:44.849203 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:30:44.849509 392 external_mini_cluster.cc:1428] Setting key a2243534821c59c9e9c565fb14a16005
Sep 26 11:30:44 dist-test-slave-mrck krb5kdc[10579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:30:44.855875 10663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:44.844448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49751 (local address 127.0.98.62:40741)
0926 11:30:44.844583 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:44.844587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:44.845539 (+ 952us) server_negotiation.cc:407] Connection header received
0926 11:30:44.846328 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:44.846330 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:44.846373 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:44.846447 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:44.847484 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:44.848121 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:44.849223 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:44.849546 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:44.849643 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:44.852189 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:44.852195 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:44.852201 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:44.852233 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:44.853737 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:44.854236 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:44.854243 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:44.854248 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:44.854343 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:44.854652 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:44.854657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:44.854658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:44.854904 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:44.855025 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:44.855484 (+ 459us) server_negotiation.cc:299] Negotiation successful
0926 11:30:44.855621 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 11:30:44.856581 10817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:44.844807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40741 (local address 127.0.98.1:49751)
0926 11:30:44.845404 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:44.845442 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:44.846153 (+ 711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:44.846597 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:44.846604 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:44.846940 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:44.847371 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:44.847382 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:44.848224 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:44.848227 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:44.849095 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:44.849104 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:44.849715 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:44.849720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:44.849911 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:44.850521 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:44.850540 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:44.852030 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:44.853862 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:44.853867 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:44.853877 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:44.854116 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:44.854432 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:44.854435 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:44.854436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:44.854556 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:44.855009 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:44.855014 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:44.855247 (+ 233us) client_negotiation.cc:769] Sending connection context
0926 11:30:44.855399 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 11:30:44.855635 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":103,"threads_started":1}
I20260926 11:30:44.857973 10815 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40741
I20260926 11:30:44.858263 10815 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:44.858778 10815 heartbeater.cc:511] Master 127.0.98.62:40741 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 11:30:44.860368 10624 ts_manager.cc:194] Registered new tserver with Master: cdb64b98fa5741e7a728e51422f4c3ff (127.0.98.1:41259)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 11:30:44.861671 10624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:30:44.876073 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40741
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:44.986819 10820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:44.987094 10820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:44.987139 10820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:44.991187 10820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:44.991253 10820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:44.991313 10820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:30:44.996898 10820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:44.997924 10820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:44.998871 10820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:45.008946 10828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:45.008947 10825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:45.009011 10826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:45.009262 10820 server_base.cc:1034] running on GCE node
I20260926 11:30:45.009819 10820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:45.010448 10820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:45.011648 10820 hybrid_clock.cc:648] HybridClock initialized: now 1790422245011629 us; error 31 us; skew 500 ppm
Sep 26 11:30:45 dist-test-slave-mrck krb5kdc[10579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:45.014395 10820 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:30:45.015333 10820 webserver.cc:460] Webserver started at http://127.0.98.2:43857/ using document root <none> and password file <none>
I20260926 11:30:45.015959 10820 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:45.016021 10820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:45.016279 10820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:45.018407 10820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-1/data/instance:
uuid: "46f77a6b6ca244b8976d83d77d4c3e44"
format_stamp: "Formatted at 2026-09-26 11:30:45 on dist-test-slave-mrck"
server_key: "90f4357d24ebbcb1fd191d663b1c4cc7"
server_key_iv: "6f682e40a5d8d0c080b79998cd90f17e"
server_key_version: "encryptionkey@0"
I20260926 11:30:45.018971 10820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "46f77a6b6ca244b8976d83d77d4c3e44"
format_stamp: "Formatted at 2026-09-26 11:30:45 on dist-test-slave-mrck"
server_key: "90f4357d24ebbcb1fd191d663b1c4cc7"
server_key_iv: "6f682e40a5d8d0c080b79998cd90f17e"
server_key_version: "encryptionkey@0"
I20260926 11:30:45.022666 10820 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 11:30:45.025074 10835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:45.026329 10820 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 11:30:45.026465 10820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "46f77a6b6ca244b8976d83d77d4c3e44"
format_stamp: "Formatted at 2026-09-26 11:30:45 on dist-test-slave-mrck"
server_key: "90f4357d24ebbcb1fd191d663b1c4cc7"
server_key_iv: "6f682e40a5d8d0c080b79998cd90f17e"
server_key_version: "encryptionkey@0"
I20260926 11:30:45.026580 10820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:45.042892 10820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:45.046039 10820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:45.046254 10820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:45.046862 10820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:45.047686 10820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:45.047803 10820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:45.047888 10820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:45.047924 10820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:45.056529 10820 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40597
I20260926 11:30:45.056552 10948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40597 every 8 connection(s)
I20260926 11:30:45.057404 10820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:30:45.062129 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 10820
I20260926 11:30:45.062220 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:30:45.062523 392 external_mini_cluster.cc:1428] Setting key bade1f570ec1969bd733374c113666ed
Sep 26 11:30:45 dist-test-slave-mrck krb5kdc[10579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:30:45.071209 10663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:45.059173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49559 (local address 127.0.98.62:40741)
0926 11:30:45.059312 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:45.059315 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:45.060455 (+ 1140us) server_negotiation.cc:407] Connection header received
0926 11:30:45.061430 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:45.061433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:45.061490 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:45.061592 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:45.062867 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:45.063385 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:45.064549 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:45.064765 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:45.064874 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:45.067496 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:45.067499 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:45.067501 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:45.067523 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:45.069079 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:45.069680 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:45.069684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:45.069686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:45.069766 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:45.070098 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:45.070103 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:45.070105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:45.070286 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:45.070386 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:45.070902 (+ 516us) server_negotiation.cc:299] Negotiation successful
0926 11:30:45.071039 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 11:30:45.072108 10951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:45.059579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40741 (local address 127.0.98.2:49559)
0926 11:30:45.060309 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:45.060350 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:45.061223 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:45.061773 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:45.061780 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:45.062150 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:45.062723 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:45.062739 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:45.063504 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:45.063510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:45.064426 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:45.064435 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:45.064906 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:45.064910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:45.065075 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:45.065724 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:45.065749 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:45.067323 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:45.069244 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:45.069250 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:45.069265 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:45.069574 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:45.069859 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:45.069863 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:45.069865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:45.070004 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:45.070394 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:45.070404 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:45.070674 (+ 270us) client_negotiation.cc:769] Sending connection context
0926 11:30:45.070864 (+ 190us) client_negotiation.cc:240] Negotiation successful
0926 11:30:45.071186 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":181,"threads_started":1}
I20260926 11:30:45.073333 10949 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40741
I20260926 11:30:45.073594 10949 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 11:30:45.074116 10949 heartbeater.cc:511] Master 127.0.98.62:40741 requested a full tablet report, sending...
I20260926 11:30:45.075152 10624 ts_manager.cc:194] Registered new tserver with Master: 46f77a6b6ca244b8976d83d77d4c3e44 (127.0.98.2:40597)
I20260926 11:30:45.075774 10624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49559
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:30:45.090773 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40741
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:30:45.202355 10954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:30:45.202627 10954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:30:45.202702 10954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:30:45.206951 10954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:30:45.207010 10954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:30:45.207091 10954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:30:45.213665 10954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:30:45.214823 10954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:30:45.215850 10954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:30:45.225517 10959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:45.225526 10960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:30:45.225533 10962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:30:45.225692 10954 server_base.cc:1034] running on GCE node
I20260926 11:30:45.226262 10954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:30:45.226948 10954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:30:45.228125 10954 hybrid_clock.cc:648] HybridClock initialized: now 1790422245228109 us; error 30 us; skew 500 ppm
Sep 26 11:30:45 dist-test-slave-mrck krb5kdc[10579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:30:45.231144 10954 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:30:45.232126 10954 webserver.cc:460] Webserver started at http://127.0.98.3:39617/ using document root <none> and password file <none>
I20260926 11:30:45.232686 10954 fs_manager.cc:359] Metadata directory not provided
I20260926 11:30:45.232782 10954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:30:45.233011 10954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:30:45.235019 10954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-2/data/instance:
uuid: "762dece6d9664710976c20a97ad3c3cf"
format_stamp: "Formatted at 2026-09-26 11:30:45 on dist-test-slave-mrck"
server_key: "3f782325728dcea985c3bb55d5418c03"
server_key_iv: "3015b3ccc6e4edaeca977b59972ef457"
server_key_version: "encryptionkey@0"
I20260926 11:30:45.235586 10954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "762dece6d9664710976c20a97ad3c3cf"
format_stamp: "Formatted at 2026-09-26 11:30:45 on dist-test-slave-mrck"
server_key: "3f782325728dcea985c3bb55d5418c03"
server_key_iv: "3015b3ccc6e4edaeca977b59972ef457"
server_key_version: "encryptionkey@0"
I20260926 11:30:45.239279 10954 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 11:30:45.241698 10969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:45.242826 10954 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 11:30:45.242964 10954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "762dece6d9664710976c20a97ad3c3cf"
format_stamp: "Formatted at 2026-09-26 11:30:45 on dist-test-slave-mrck"
server_key: "3f782325728dcea985c3bb55d5418c03"
server_key_iv: "3015b3ccc6e4edaeca977b59972ef457"
server_key_version: "encryptionkey@0"
I20260926 11:30:45.243067 10954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:30:45.268749 10954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:30:45.272224 10954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:30:45.272436 10954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:30:45.273061 10954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:30:45.273954 10954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:30:45.274025 10954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:45.274098 10954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:30:45.274140 10954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:30:45.282902 10954 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40025
I20260926 11:30:45.282979 11082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40025 every 8 connection(s)
I20260926 11:30:45.283839 10954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:30:45.287478 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 10954
I20260926 11:30:45.287600 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:30:45.287950 392 external_mini_cluster.cc:1428] Setting key 1552090f58a7e483afe9917fff6ba629
Sep 26 11:30:45 dist-test-slave-mrck krb5kdc[10579](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:30:45.297528 10663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:45.285588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59085 (local address 127.0.98.62:40741)
0926 11:30:45.285722 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:45.285727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:45.286946 (+ 1219us) server_negotiation.cc:407] Connection header received
0926 11:30:45.288085 (+ 1139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:45.288088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:45.288145 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:45.288211 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:45.289728 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:45.290337 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:45.291471 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:45.291685 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:45.291873 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:45.294227 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:45.294230 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:45.294233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:45.294264 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:45.295482 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:45.296054 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:45.296059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:45.296061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:45.296142 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:45.296488 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:45.296493 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:45.296495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:45.296677 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:45.296785 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:45.297258 (+ 473us) server_negotiation.cc:299] Negotiation successful
0926 11:30:45.297373 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 11:30:45.298339 11085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:45.285960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40741 (local address 127.0.98.3:59085)
0926 11:30:45.286789 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:45.286830 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:45.287840 (+ 1010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:45.288445 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:45.288455 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:45.288960 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:30:45.289573 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:45.289586 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:45.290461 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:45.290465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:45.291341 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:45.291350 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:45.291909 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:45.291914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:45.292091 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:45.292627 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:30:45.292643 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:30:45.294066 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:30:45.295595 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:45.295601 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:45.295615 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:45.295952 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:45.296250 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:30:45.296253 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:30:45.296255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:30:45.296393 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:30:45.296781 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:30:45.296787 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:30:45.297032 (+ 245us) client_negotiation.cc:769] Sending connection context
0926 11:30:45.297227 (+ 195us) client_negotiation.cc:240] Negotiation successful
0926 11:30:45.297543 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":516,"threads_started":1}
I20260926 11:30:45.299455 11083 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40741
I20260926 11:30:45.299707 11083 heartbeater.cc:464] Registering TS with master...
I20260926 11:30:45.300272 11083 heartbeater.cc:511] Master 127.0.98.62:40741 requested a full tablet report, sending...
I20260926 11:30:45.301441 10624 ts_manager.cc:194] Registered new tserver with Master: 762dece6d9664710976c20a97ad3c3cf (127.0.98.3:40025)
I20260926 11:30:45.301914 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:30:45.302191 10624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59085
I20260926 11:30:45.310781 10663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:45.303975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48436 (local address 127.0.98.62:40741)
0926 11:30:45.304098 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:45.304102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:45.304293 (+ 191us) server_negotiation.cc:407] Connection header received
0926 11:30:45.304380 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:45.304382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:45.304418 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:45.304485 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:30:45.305173 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:45.305704 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:45.306497 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:45.306725 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:45.306814 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:45.308223 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:30:45.308226 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:30:45.308229 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:30:45.308249 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:30:45.309455 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:45.309821 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:45.309824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:45.309825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:45.309896 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:30:45.310107 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:30:45.310109 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:30:45.310110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:30:45.310311 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:30:45.310409 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:45.310583 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 11:30:45.310671 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 11:30:45.313848 10624 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:30:45.316910 10624 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:30:45.332260 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:45.325227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36612 (local address 127.0.98.1:41259)
0926 11:30:45.326331 (+ 1104us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:45.326347 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:45.326370 (+ 23us) server_negotiation.cc:407] Connection header received
0926 11:30:45.326452 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:45.326459 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:45.326688 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:45.327176 (+ 488us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:30:45.327355 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:45.328580 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:45.330716 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:45.331494 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:45.331620 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:45.331813 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:45.331983 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 11:30:45.332082 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":988,"thread_start_us":86,"threads_started":1}
I20260926 11:30:45.332252 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:45.325109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41259 (local address 127.0.0.1:36612)
0926 11:30:45.325574 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:45.325622 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:45.325764 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:45.326865 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:45.326871 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:45.326894 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:30:45.327039 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:45.327049 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:45.328730 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:45.328734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:45.330581 (+ 1847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:45.330594 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:45.331618 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:45.331622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:45.331773 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:45.331816 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 11:30:45.331954 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 11:30:45.332086 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":409,"thread_start_us":95,"threads_started":1}
I20260926 11:30:45.332275 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:45.325827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40597 (local address 127.0.0.1:49776)
0926 11:30:45.326106 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:45.326119 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:45.326216 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:45.327092 (+ 876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:45.327094 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:45.327106 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:30:45.327229 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:45.327235 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:45.328730 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:45.328734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:45.330748 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:45.330760 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:45.331832 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:45.331837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:45.331925 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:45.331961 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 11:30:45.332013 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 11:30:45.332091 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":229,"thread_start_us":85,"threads_started":1}
I20260926 11:30:45.332650 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:45.325504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40025 (local address 127.0.0.1:50732)
0926 11:30:45.325743 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:45.325757 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:45.325851 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:45.327022 (+ 1171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:45.327031 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:45.327068 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:30:45.327229 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:45.327235 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:45.329893 (+ 2658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:45.329899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:45.331009 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:45.331019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:45.332216 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:45.332219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:45.332317 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:45.332349 (+ 32us) client_negotiation.cc:769] Sending connection context
0926 11:30:45.332424 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 11:30:45.332512 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":44,"thread_start_us":109,"threads_started":1}
I20260926 11:30:45.333122 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:45.325945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50732 (local address 127.0.98.3:40025)
0926 11:30:45.326363 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:45.326374 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:45.326396 (+ 22us) server_negotiation.cc:407] Connection header received
0926 11:30:45.326475 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:45.326479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:45.326681 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:45.326842 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:30:45.328766 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:45.329757 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:45.331111 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:45.331919 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:45.332022 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:45.332157 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:45.332885 (+ 728us) server_negotiation.cc:299] Negotiation successful
0926 11:30:45.332986 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":114,"threads_started":1}
I20260926 11:30:45.333284 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:45.326125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49776 (local address 127.0.98.2:40597)
0926 11:30:45.326696 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:45.326707 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:45.326727 (+ 20us) server_negotiation.cc:407] Connection header received
0926 11:30:45.326800 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:45.326806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:45.326964 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:45.327112 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:30:45.327347 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:45.328580 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:45.330860 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:45.331658 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:45.332775 (+ 1117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:45.332934 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:45.332977 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 11:30:45.333055 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":136,"threads_started":1}
I20260926 11:30:45.335654 11017 tablet_service.cc:1467] Processing CreateTablet for tablet 91a2f85bbbde411d88273107b6874789 (DEFAULT_TABLE table=test-table [id=67a41b3e85e14a83a363faba1a06ebc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:30:45.335654 10749 tablet_service.cc:1467] Processing CreateTablet for tablet 91a2f85bbbde411d88273107b6874789 (DEFAULT_TABLE table=test-table [id=67a41b3e85e14a83a363faba1a06ebc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:30:45.336817 10749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a2f85bbbde411d88273107b6874789. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:45.336823 11017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a2f85bbbde411d88273107b6874789. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:45.338706 10883 tablet_service.cc:1467] Processing CreateTablet for tablet 91a2f85bbbde411d88273107b6874789 (DEFAULT_TABLE table=test-table [id=67a41b3e85e14a83a363faba1a06ebc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:30:45.339843 10883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a2f85bbbde411d88273107b6874789. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:30:45.343209 11100 tablet_bootstrap.cc:492] T 91a2f85bbbde411d88273107b6874789 P cdb64b98fa5741e7a728e51422f4c3ff: Bootstrap starting.
I20260926 11:30:45.344745 11101 tablet_bootstrap.cc:492] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf: Bootstrap starting.
I20260926 11:30:45.345409 11100 tablet_bootstrap.cc:654] T 91a2f85bbbde411d88273107b6874789 P cdb64b98fa5741e7a728e51422f4c3ff: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:45.346530 11100 log.cc:824] T 91a2f85bbbde411d88273107b6874789 P cdb64b98fa5741e7a728e51422f4c3ff: Log is configured to *not* fsync() on all Append() calls
I20260926 11:30:45.347034 11101 tablet_bootstrap.cc:654] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:45.347024 11102 tablet_bootstrap.cc:492] T 91a2f85bbbde411d88273107b6874789 P 46f77a6b6ca244b8976d83d77d4c3e44: Bootstrap starting.
I20260926 11:30:45.348483 11101 log.cc:824] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf: Log is configured to *not* fsync() on all Append() calls
I20260926 11:30:45.349570 11100 tablet_bootstrap.cc:492] T 91a2f85bbbde411d88273107b6874789 P cdb64b98fa5741e7a728e51422f4c3ff: No bootstrap required, opened a new log
I20260926 11:30:45.349583 11102 tablet_bootstrap.cc:654] T 91a2f85bbbde411d88273107b6874789 P 46f77a6b6ca244b8976d83d77d4c3e44: Neither blocks nor log segments found. Creating new log.
I20260926 11:30:45.349769 11100 ts_tablet_manager.cc:1397] T 91a2f85bbbde411d88273107b6874789 P cdb64b98fa5741e7a728e51422f4c3ff: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 11:30:45.350286 11102 log.cc:824] T 91a2f85bbbde411d88273107b6874789 P 46f77a6b6ca244b8976d83d77d4c3e44: Log is configured to *not* fsync() on all Append() calls
I20260926 11:30:45.350454 11101 tablet_bootstrap.cc:492] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf: No bootstrap required, opened a new log
I20260926 11:30:45.350636 11101 ts_tablet_manager.cc:1397] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 11:30:45.351981 11102 tablet_bootstrap.cc:492] T 91a2f85bbbde411d88273107b6874789 P 46f77a6b6ca244b8976d83d77d4c3e44: No bootstrap required, opened a new log
I20260926 11:30:45.352177 11102 ts_tablet_manager.cc:1397] T 91a2f85bbbde411d88273107b6874789 P 46f77a6b6ca244b8976d83d77d4c3e44: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 11:30:45.353819 11100 raft_consensus.cc:348] T 91a2f85bbbde411d88273107b6874789 P cdb64b98fa5741e7a728e51422f4c3ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb64b98fa5741e7a728e51422f4c3ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41259 } } peers { permanent_uuid: "46f77a6b6ca244b8976d83d77d4c3e44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40597 } } peers { permanent_uuid: "762dece6d9664710976c20a97ad3c3cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260926 11:30:45.354421 11101 raft_consensus.cc:348] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb64b98fa5741e7a728e51422f4c3ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41259 } } peers { permanent_uuid: "46f77a6b6ca244b8976d83d77d4c3e44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40597 } } peers { permanent_uuid: "762dece6d9664710976c20a97ad3c3cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260926 11:30:45.354709 11101 raft_consensus.cc:374] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:45.354717 11100 raft_consensus.cc:374] T 91a2f85bbbde411d88273107b6874789 P cdb64b98fa5741e7a728e51422f4c3ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:45.354780 11100 raft_consensus.cc:729] T 91a2f85bbbde411d88273107b6874789 P cdb64b98fa5741e7a728e51422f4c3ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdb64b98fa5741e7a728e51422f4c3ff, State: Initialized, Role: FOLLOWER
I20260926 11:30:45.354777 11101 raft_consensus.cc:729] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 762dece6d9664710976c20a97ad3c3cf, State: Initialized, Role: FOLLOWER
I20260926 11:30:45.355103 11102 raft_consensus.cc:348] T 91a2f85bbbde411d88273107b6874789 P 46f77a6b6ca244b8976d83d77d4c3e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb64b98fa5741e7a728e51422f4c3ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41259 } } peers { permanent_uuid: "46f77a6b6ca244b8976d83d77d4c3e44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40597 } } peers { permanent_uuid: "762dece6d9664710976c20a97ad3c3cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260926 11:30:45.355268 11100 consensus_queue.cc:260] T 91a2f85bbbde411d88273107b6874789 P cdb64b98fa5741e7a728e51422f4c3ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb64b98fa5741e7a728e51422f4c3ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41259 } } peers { permanent_uuid: "46f77a6b6ca244b8976d83d77d4c3e44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40597 } } peers { permanent_uuid: "762dece6d9664710976c20a97ad3c3cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260926 11:30:45.355268 11101 consensus_queue.cc:260] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb64b98fa5741e7a728e51422f4c3ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41259 } } peers { permanent_uuid: "46f77a6b6ca244b8976d83d77d4c3e44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40597 } } peers { permanent_uuid: "762dece6d9664710976c20a97ad3c3cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260926 11:30:45.355383 11102 raft_consensus.cc:374] T 91a2f85bbbde411d88273107b6874789 P 46f77a6b6ca244b8976d83d77d4c3e44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:30:45.355450 11102 raft_consensus.cc:729] T 91a2f85bbbde411d88273107b6874789 P 46f77a6b6ca244b8976d83d77d4c3e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46f77a6b6ca244b8976d83d77d4c3e44, State: Initialized, Role: FOLLOWER
I20260926 11:30:45.356021 11083 heartbeater.cc:503] Master 127.0.98.62:40741 was elected leader, sending a full tablet report...
I20260926 11:30:45.356281 10815 heartbeater.cc:503] Master 127.0.98.62:40741 was elected leader, sending a full tablet report...
I20260926 11:30:45.356365 11101 ts_tablet_manager.cc:1428] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 11:30:45.356366 11100 ts_tablet_manager.cc:1428] T 91a2f85bbbde411d88273107b6874789 P cdb64b98fa5741e7a728e51422f4c3ff: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 11:30:45.357163 11102 consensus_queue.cc:260] T 91a2f85bbbde411d88273107b6874789 P 46f77a6b6ca244b8976d83d77d4c3e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb64b98fa5741e7a728e51422f4c3ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41259 } } peers { permanent_uuid: "46f77a6b6ca244b8976d83d77d4c3e44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40597 } } peers { permanent_uuid: "762dece6d9664710976c20a97ad3c3cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260926 11:30:45.357815 10949 heartbeater.cc:503] Master 127.0.98.62:40741 was elected leader, sending a full tablet report...
I20260926 11:30:45.358266 11102 ts_tablet_manager.cc:1428] T 91a2f85bbbde411d88273107b6874789 P 46f77a6b6ca244b8976d83d77d4c3e44: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 11:30:45.374154 11106 raft_consensus.cc:482] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:30:45.374332 11106 raft_consensus.cc:504] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb64b98fa5741e7a728e51422f4c3ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41259 } } peers { permanent_uuid: "46f77a6b6ca244b8976d83d77d4c3e44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40597 } } peers { permanent_uuid: "762dece6d9664710976c20a97ad3c3cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260926 11:30:45.375351 11106 leader_election.cc:288] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdb64b98fa5741e7a728e51422f4c3ff (127.0.98.1:41259), 46f77a6b6ca244b8976d83d77d4c3e44 (127.0.98.2:40597)
I20260926 11:30:45.378875 11085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:45.375610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41259 (local address 127.0.98.3:48377)
0926 11:30:45.375718 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:45.375759 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:45.375856 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:45.376131 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:45.376134 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:45.376154 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:30:45.376270 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:45.376276 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:45.377003 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:45.377009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:45.377798 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:45.377807 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:45.378442 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:45.378445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:45.378564 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:45.378582 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 11:30:45.378674 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 11:30:45.378745 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 11:30:45.378875 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:45.375687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48377 (local address 127.0.98.1:41259)
0926 11:30:45.375850 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:45.375853 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:45.375887 (+ 34us) server_negotiation.cc:407] Connection header received
0926 11:30:45.375949 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:45.375954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:45.376013 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:45.376109 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:30:45.376388 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:45.376892 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:45.377929 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:45.378322 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:45.378442 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:45.378507 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:45.378690 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 11:30:45.378761 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 11:30:45.379585 11099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:45.376053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34721 (local address 127.0.98.2:40597)
0926 11:30:45.376181 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:45.376188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:45.376426 (+ 238us) server_negotiation.cc:407] Connection header received
0926 11:30:45.376542 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:45.376545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:45.376594 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:45.376681 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:30:45.376949 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:45.377442 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:45.378469 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:45.378875 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:45.379170 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:45.379244 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:45.379429 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 11:30:45.379491 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 11:30:45.379594 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:45.375934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40597 (local address 127.0.98.3:34721)
0926 11:30:45.376333 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:45.376343 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 11:30:45.376453 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:30:45.376687 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:30:45.376690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:30:45.376706 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:30:45.376838 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:45.376844 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:45.377554 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:45.377560 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:45.378342 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:30:45.378350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:30:45.379199 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:30:45.379204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:30:45.379283 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:45.379300 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 11:30:45.379400 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 11:30:45.379460 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":12,"thread_start_us":112,"threads_started":1}
I20260926 11:30:45.379635 10769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a2f85bbbde411d88273107b6874789" candidate_uuid: "762dece6d9664710976c20a97ad3c3cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdb64b98fa5741e7a728e51422f4c3ff" is_pre_election: true
I20260926 11:30:45.379945 10769 raft_consensus.cc:2454] T 91a2f85bbbde411d88273107b6874789 P cdb64b98fa5741e7a728e51422f4c3ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 762dece6d9664710976c20a97ad3c3cf in term 0.
I20260926 11:30:45.380398 10903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a2f85bbbde411d88273107b6874789" candidate_uuid: "762dece6d9664710976c20a97ad3c3cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46f77a6b6ca244b8976d83d77d4c3e44" is_pre_election: true
I20260926 11:30:45.380457 10972 leader_election.cc:302] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 762dece6d9664710976c20a97ad3c3cf, cdb64b98fa5741e7a728e51422f4c3ff; no voters:
I20260926 11:30:45.380702 10903 raft_consensus.cc:2454] T 91a2f85bbbde411d88273107b6874789 P 46f77a6b6ca244b8976d83d77d4c3e44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 762dece6d9664710976c20a97ad3c3cf in term 0.
I20260926 11:30:45.380740 11106 raft_consensus.cc:2781] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:30:45.380807 11106 raft_consensus.cc:482] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:30:45.380837 11106 raft_consensus.cc:3037] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:45.381944 11106 raft_consensus.cc:504] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb64b98fa5741e7a728e51422f4c3ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41259 } } peers { permanent_uuid: "46f77a6b6ca244b8976d83d77d4c3e44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40597 } } peers { permanent_uuid: "762dece6d9664710976c20a97ad3c3cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260926 11:30:45.382326 11106 leader_election.cc:288] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf [CANDIDATE]: Term 1 election: Requested vote from peers cdb64b98fa5741e7a728e51422f4c3ff (127.0.98.1:41259), 46f77a6b6ca244b8976d83d77d4c3e44 (127.0.98.2:40597)
I20260926 11:30:45.382661 10769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a2f85bbbde411d88273107b6874789" candidate_uuid: "762dece6d9664710976c20a97ad3c3cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdb64b98fa5741e7a728e51422f4c3ff"
I20260926 11:30:45.382763 10903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a2f85bbbde411d88273107b6874789" candidate_uuid: "762dece6d9664710976c20a97ad3c3cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46f77a6b6ca244b8976d83d77d4c3e44"
I20260926 11:30:45.382870 10769 raft_consensus.cc:3037] T 91a2f85bbbde411d88273107b6874789 P cdb64b98fa5741e7a728e51422f4c3ff [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:45.382902 10903 raft_consensus.cc:3037] T 91a2f85bbbde411d88273107b6874789 P 46f77a6b6ca244b8976d83d77d4c3e44 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:30:45.383975 10769 raft_consensus.cc:2454] T 91a2f85bbbde411d88273107b6874789 P cdb64b98fa5741e7a728e51422f4c3ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 762dece6d9664710976c20a97ad3c3cf in term 1.
I20260926 11:30:45.384048 10903 raft_consensus.cc:2454] T 91a2f85bbbde411d88273107b6874789 P 46f77a6b6ca244b8976d83d77d4c3e44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 762dece6d9664710976c20a97ad3c3cf in term 1.
I20260926 11:30:45.384315 10972 leader_election.cc:302] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 762dece6d9664710976c20a97ad3c3cf, cdb64b98fa5741e7a728e51422f4c3ff; no voters:
I20260926 11:30:45.384488 11106 raft_consensus.cc:2781] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:30:45.384716 11106 raft_consensus.cc:686] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf [term 1 LEADER]: Becoming Leader. State: Replica: 762dece6d9664710976c20a97ad3c3cf, State: Running, Role: LEADER
I20260926 11:30:45.385062 11106 consensus_queue.cc:237] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb64b98fa5741e7a728e51422f4c3ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41259 } } peers { permanent_uuid: "46f77a6b6ca244b8976d83d77d4c3e44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40597 } } peers { permanent_uuid: "762dece6d9664710976c20a97ad3c3cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } }
I20260926 11:30:45.388271 10621 catalog_manager.cc:5495] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf reported cstate change: term changed from 0 to 1, leader changed from <none> to 762dece6d9664710976c20a97ad3c3cf (127.0.98.3). New cstate: current_term: 1 leader_uuid: "762dece6d9664710976c20a97ad3c3cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdb64b98fa5741e7a728e51422f4c3ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46f77a6b6ca244b8976d83d77d4c3e44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "762dece6d9664710976c20a97ad3c3cf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40025 } health_report { overall_health: HEALTHY } } }
I20260926 11:30:45.411369 11098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:45.407824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50746 (local address 127.0.98.3:40025)
0926 11:30:45.407952 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 11:30:45.407956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:30:45.407971 (+ 15us) server_negotiation.cc:407] Connection header received
0926 11:30:45.408143 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:30:45.408146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:30:45.408203 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:30:45.408317 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:30:45.408554 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:45.409058 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:45.409908 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:30:45.410143 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:30:45.410256 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:30:45.410561 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:30:45.410985 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:30:45.411127 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 11:30:45.411168 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 11:30:45.411223 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 11:30:45.417238 10903 raft_consensus.cc:1261] T 91a2f85bbbde411d88273107b6874789 P 46f77a6b6ca244b8976d83d77d4c3e44 [term 1 FOLLOWER]: Refusing update from remote peer 762dece6d9664710976c20a97ad3c3cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:30:45.417363 10769 raft_consensus.cc:1261] T 91a2f85bbbde411d88273107b6874789 P cdb64b98fa5741e7a728e51422f4c3ff [term 1 FOLLOWER]: Refusing update from remote peer 762dece6d9664710976c20a97ad3c3cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:30:45.418066 11106 consensus_queue.cc:1035] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "46f77a6b6ca244b8976d83d77d4c3e44" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:30:45.418304 11110 consensus_queue.cc:1035] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdb64b98fa5741e7a728e51422f4c3ff" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:30:45.426738 11114 mvcc.cc:205] Tried to move back new op lower bound from 7333569517221408768 to 7333569517100167168. Current Snapshot: MvccSnapshot[applied={T|T < 7333569517221408768}]
I20260926 11:30:45.427851 11115 mvcc.cc:205] Tried to move back new op lower bound from 7333569517221408768 to 7333569517100167168. Current Snapshot: MvccSnapshot[applied={T|T < 7333569517221408768}]
I20260926 11:30:45.433389 10621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:30:45.433565 10621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:30:45.435847 10621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a0be1489984040438a61f0f4e07751c7: Sending DeleteTablet for 3 replicas of tablet 91a2f85bbbde411d88273107b6874789
I20260926 11:30:45.436735 11017 tablet_service.cc:1514] Processing DeleteTablet for tablet 91a2f85bbbde411d88273107b6874789 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:30:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50732
I20260926 11:30:45.436857 10749 tablet_service.cc:1514] Processing DeleteTablet for tablet 91a2f85bbbde411d88273107b6874789 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:30:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36612
I20260926 11:30:45.437256 11122 tablet_replica.cc:330] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf: stopping tablet replica
I20260926 11:30:45.437445 11123 tablet_replica.cc:330] T 91a2f85bbbde411d88273107b6874789 P cdb64b98fa5741e7a728e51422f4c3ff: stopping tablet replica
I20260926 11:30:45.437419 10883 tablet_service.cc:1514] Processing DeleteTablet for tablet 91a2f85bbbde411d88273107b6874789 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:30:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49776
I20260926 11:30:45.437587 11122 raft_consensus.cc:2229] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:30:45.437880 11123 raft_consensus.cc:2229] T 91a2f85bbbde411d88273107b6874789 P cdb64b98fa5741e7a728e51422f4c3ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:30:45.438222 11122 raft_consensus.cc:2258] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:30:45.438244 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 10686
I20260926 11:30:45.439216 11124 tablet_replica.cc:330] T 91a2f85bbbde411d88273107b6874789 P 46f77a6b6ca244b8976d83d77d4c3e44: stopping tablet replica
I20260926 11:30:45.439610 11124 raft_consensus.cc:2229] T 91a2f85bbbde411d88273107b6874789 P 46f77a6b6ca244b8976d83d77d4c3e44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:30:45.440044 11124 raft_consensus.cc:2258] T 91a2f85bbbde411d88273107b6874789 P 46f77a6b6ca244b8976d83d77d4c3e44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:30:45.441747 11124 ts_tablet_manager.cc:1905] T 91a2f85bbbde411d88273107b6874789 P 46f77a6b6ca244b8976d83d77d4c3e44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:30:45.445282 11124 ts_tablet_manager.cc:1918] T 91a2f85bbbde411d88273107b6874789 P 46f77a6b6ca244b8976d83d77d4c3e44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260926 11:30:45.445273 10610 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41259 (error 108)
I20260926 11:30:45.445386 11124 log.cc:1196] T 91a2f85bbbde411d88273107b6874789 P 46f77a6b6ca244b8976d83d77d4c3e44: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-1/wal/wals/91a2f85bbbde411d88273107b6874789
I20260926 11:30:45.445685 11122 ts_tablet_manager.cc:1905] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:30:45.445749 11124 ts_tablet_manager.cc:1939] T 91a2f85bbbde411d88273107b6874789 P 46f77a6b6ca244b8976d83d77d4c3e44: Deleting consensus metadata
I20260926 11:30:45.446112 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 10820
I20260926 11:30:45.448702 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:45.448327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41259 (local address 127.0.0.1:36628)
0926 11:30:45.448528 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:45.448604 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260926 11:30:45.448921 10610 catalog_manager.cc:4584] TS cdb64b98fa5741e7a728e51422f4c3ff (127.0.98.1:41259): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 91a2f85bbbde411d88273107b6874789: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41259: connect: Connection refused (error 111)
I20260926 11:30:45.449961 11122 ts_tablet_manager.cc:1918] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:30:45.450089 11122 log.cc:1196] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790422180210734-392-0/minicluster-data/ts-2/wal/wals/91a2f85bbbde411d88273107b6874789
I20260926 11:30:45.450402 11122 ts_tablet_manager.cc:1939] T 91a2f85bbbde411d88273107b6874789 P 762dece6d9664710976c20a97ad3c3cf: Deleting consensus metadata
I20260926 11:30:45.451489 10608 catalog_manager.cc:4841] TS 762dece6d9664710976c20a97ad3c3cf (127.0.98.3:40025): tablet 91a2f85bbbde411d88273107b6874789 (table test-table [id=67a41b3e85e14a83a363faba1a06ebc4]) successfully deleted
I20260926 11:30:45.453387 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 10954
I20260926 11:30:45.453567 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:30:45.453292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40597 (local address 127.0.0.1:49792)
0926 11:30:45.453441 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 11:30:45.453494 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 11:30:45.458714 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 10592
2026-09-26T11:30:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2198 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:30:45 dist-test-slave-mrck krb5kdc[11128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:30:45 dist-test-slave-mrck krb5kdc[11128](info): set up 2 sockets
Sep 26 11:30:45 dist-test-slave-mrck krb5kdc[11128](info): commencing operation
krb5kdc: starting...
W20260926 11:30:46.497066 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:30:46 dist-test-slave-mrck krb5kdc[11128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:30:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:30:46Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260926 11:30:46.662756 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskF2HClO/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/postgres -l logfile start
2026-09-26 11:30:49.102 UTC [11158] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 11:30:49.102 UTC [11158] LOG: listening on IPv4 address "127.0.98.20", port 43473
2026-09-26 11:30:49.106 UTC [11158] LOG: listening on Unix socket "/tmp/.s.PGSQL.43473"
2026-09-26 11:30:49.128 UTC [11161] LOG: database system was shut down at 2026-09-26 11:30:47 UTC
2026-09-26 11:30:49.132 UTC [11158] LOG: database system is ready to accept connections
I20260926 11:30:50.088891 392 mini_postgres.cc:96] Postgres bound to 43473
2026-09-26 11:30:50.095 UTC [11169] FATAL: database "slave" does not exist
127.0.98.20:43473 - accepting connections
I20260926 11:30:50.096002 392 mini_ranger.cc:161] Starting Ranger...
I20260926 11:30:50.111469 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 11:30:50.594187 392 mini_ranger.cc:87] Created ranger Postgres database
I20260926 11:30:50.594269 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ranger-admin
2026-09-26 11:30:50,882 [I] DB FLAVOR :POSTGRES
2026-09-26 11:30:50,882 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 11:30:50,883 [I] Checking connection..
2026-09-26 11:30:50,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 11:30:51,132 [I] Checking connection passed.
2026-09-26 11:30:51,133 [I] --------- Verifying version history table ---------
2026-09-26 11:30:51,133 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 11:30:51,397 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 11:30:51,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 11:30:51,661 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 11:30:51,661 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 11:30:51,661 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43473/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 11:30:51.895 UTC [11233] WARNING: there is no transaction in progress
2026-09-26 11:30:51,902 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 11:30:51,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 11:30:52,130 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 11:30:52,130 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 11:30:52,130 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 11:30:52,380 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 11:30:52,638 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-mrck.c.gcp-upstream.internal','N') ;"
2026-09-26 11:30:52,868 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 11:30:53,122 [I] Table x_portal_user does not exist in database ranger
2026-09-26 11:30:53,123 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 11:30:53,379 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 11:30:53,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 11:30:53,617 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 11:30:53,853 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 11:30:53,853 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43473/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 11:30:54.776 UTC [11341] WARNING: there is no transaction in progress
2026-09-26 11:30:54.793 UTC [11341] WARNING: there is no transaction in progress
2026-09-26 11:30:54.809 UTC [11341] WARNING: there is no transaction in progress
2026-09-26 11:30:54.824 UTC [11341] WARNING: there is no transaction in progress
2026-09-26 11:30:54.839 UTC [11341] WARNING: there is no transaction in progress
2026-09-26 11:30:54.855 UTC [11341] WARNING: there is no transaction in progress
2026-09-26 11:30:54.911 UTC [11341] WARNING: there is no transaction in progress
2026-09-26 11:30:54.918 UTC [11341] WARNING: there is no transaction in progress
2026-09-26 11:30:54.932 UTC [11341] WARNING: there is no transaction in progress
2026-09-26 11:30:54.943 UTC [11341] WARNING: there is no transaction in progress
2026-09-26 11:30:54.953 UTC [11341] WARNING: there is no transaction in progress
2026-09-26 11:30:54.967 UTC [11341] WARNING: there is no transaction in progress
2026-09-26 11:30:54.977 UTC [11341] WARNING: there is no transaction in progress
2026-09-26 11:30:54.983 UTC [11341] WARNING: there is no transaction in progress
2026-09-26 11:30:54.990 UTC [11341] WARNING: there is no transaction in progress
2026-09-26 11:30:54.997 UTC [11341] WARNING: there is no transaction in progress
2026-09-26 11:30:55,621 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 11:30:55,621 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 11:30:55,854 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 11:30:55,855 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 11:30:56,085 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 11:30:56,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 11:30:56,331 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 11:30:56,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 11:30:56,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 11:30:56,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-mrck.c.gcp-upstream.internal';"
2026-09-26 11:30:57,077 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 11:30:57,078 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43473/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 11:30:57,331 [I] DB_PATCHES have already been applied
I20260926 11:30:57.337230 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskF2HClO/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 11:30:57.337301 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260926 11:30:57.340674 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:34717
Sep 26, 2026 11:30:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 11:30:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 11:30:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 11:30:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 11:30:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 11:30:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 11:30:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 11:30:58.460712 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.365s user 0.000s sys 0.010s
Sep 26, 2026 11:31:17 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 11:31:17 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 11:31:17 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 11:31:17 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 11:31:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 11:31:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 11:31:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 11:31:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 11:31:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 11:31:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 11:31:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 11:31:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 11:31:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 11:31:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 11:31:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 11:31:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 11:31:17 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 11:31:18 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 11:31:19.383622 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 11:31:19 dist-test-slave-mrck krb5kdc[11128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422279, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 11:31:19.469079 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 11:31:19.479058 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 11:31:20.350543 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 11:31:20.350667 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ranger-kms
I20260926 11:31:20.350776 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:43473
2026-09-26 11:31:20,534 [I] DB FLAVOR :POSTGRES
2026-09-26 11:31:20,534 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 11:31:20,534 [I] Checking connection
2026-09-26 11:31:20,534 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43473/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 11:31:20,806 [I] connection success
2026-09-26 11:31:20,807 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 11:31:20,807 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 11:31:20,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43473/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 11:31:21,049 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 11:31:21,049 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 11:31:21,049 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 11:31:21,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:43473/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 11:31:21,299 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 11:31:21.766760 392 mini_ranger_kms.cc:310] Created kms service
I20260926 11:31:21.954921 392 mini_ranger_kms.cc:326] Created kudu user
I20260926 11:31:22.002827 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 11:31:22 dist-test-slave-mrck krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422279, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:31:22.441541 392 mini_ranger_kms.cc:384] Added ranger policy
I20260926 11:31:22.441696 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 11:31:22.441717 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260926 11:31:22.445511 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11521
I20260926 11:31:22.445653 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:60815
Sep 26, 2026 11:31:23 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 11:31:23 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 11:31:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 11:31:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 11:31:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 11:31:24 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 11:31:24.751457 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.282s user 0.015s sys 0.152s
I20260926 11:31:24.751607 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 11:31:24.751751 392 mini_ranger_kms.cc:401] 127.0.98.20:60815/kms/v1/keys
Sep 26 11:31:24 dist-test-slave-mrck krb5kdc[11128](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790422284, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 11:31:25 dist-test-slave-mrck krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422284, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 11:31:25 dist-test-slave-mrck krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422284, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 11:31:27 dist-test-slave-mrck krb5kdc[11128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:31:28.018951 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45275
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45275
--ranger_config_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60815/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:31:28.131006 11572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:31:28.131309 11572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:31:28.131387 11572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:31:28.135550 11572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:31:28.135640 11572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:31:28.135675 11572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:31:28.135712 11572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:31:28.135757 11572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:31:28.141306 11572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:60815/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45275
--ranger_config_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:31:28.142452 11572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:31:28.143509 11572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 11:31:28.152634 11572 server_base.cc:1034] running on GCE node
W20260926 11:31:28.152577 11578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:31:28.152586 11577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:31:28.152577 11580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:31:28.153486 11572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:31:28.154592 11572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:31:28.155786 11572 hybrid_clock.cc:648] HybridClock initialized: now 1790422288155768 us; error 31 us; skew 500 ppm
Sep 26 11:31:28 dist-test-slave-mrck krb5kdc[11128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:31:28.162333 11572 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:31:28.163420 11572 webserver.cc:460] Webserver started at http://127.0.98.62:35785/ using document root <none> and password file <none>
I20260926 11:31:28.164122 11572 fs_manager.cc:359] Metadata directory not provided
I20260926 11:31:28.164188 11572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:31:28.164434 11572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 11:31:28 dist-test-slave-mrck krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422288, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:31:28.231308 11572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "2c48bf885dae44af886f283909fff17b"
format_stamp: "Formatted at 2026-09-26 11:31:28 on dist-test-slave-mrck"
server_key: "3d6814d5840ea711e0ba12a2177e87c5"
server_key_iv: "cd137f020671fabc1fb4d0f6a349379b"
server_key_version: "kuduclusterkey@0"
I20260926 11:31:28.232092 11572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "2c48bf885dae44af886f283909fff17b"
format_stamp: "Formatted at 2026-09-26 11:31:28 on dist-test-slave-mrck"
server_key: "3d6814d5840ea711e0ba12a2177e87c5"
server_key_iv: "cd137f020671fabc1fb4d0f6a349379b"
server_key_version: "kuduclusterkey@0"
I20260926 11:31:28.236593 11572 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260926 11:31:28.254536 11587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:31:28.256119 11572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:31:28.256274 11572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "2c48bf885dae44af886f283909fff17b"
format_stamp: "Formatted at 2026-09-26 11:31:28 on dist-test-slave-mrck"
server_key: "3d6814d5840ea711e0ba12a2177e87c5"
server_key_iv: "cd137f020671fabc1fb4d0f6a349379b"
server_key_version: "kuduclusterkey@0"
I20260926 11:31:28.256407 11572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:31:28.282325 11572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:31:28.285746 11572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:31:28.285976 11572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:31:28.294906 11572 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45275
I20260926 11:31:28.294946 11639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45275 every 8 connection(s)
I20260926 11:31:28.296078 11572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:31:28.296793 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 11572
I20260926 11:31:28.296937 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:31:28.297225 392 external_mini_cluster.cc:1428] Setting key 17423effae248d3bca9038883d54adef
I20260926 11:31:28.299597 11640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 11:31:28 dist-test-slave-mrck krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:31:28.307168 11640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c48bf885dae44af886f283909fff17b: Bootstrap starting.
I20260926 11:31:28.310137 11640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c48bf885dae44af886f283909fff17b: Neither blocks nor log segments found. Creating new log.
I20260926 11:31:28.311542 11640 log.cc:824] T 00000000000000000000000000000000 P 2c48bf885dae44af886f283909fff17b: Log is configured to *not* fsync() on all Append() calls
I20260926 11:31:28.311710 11643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:31:28.298840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33698 (local address 127.0.98.62:45275)
0926 11:31:28.299250 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0926 11:31:28.299271 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 11:31:28.299319 (+ 48us) server_negotiation.cc:407] Connection header received
0926 11:31:28.300082 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:31:28.300103 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:31:28.300465 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:31:28.300816 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:31:28.301633 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:31:28.302715 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:31:28.303596 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:31:28.304152 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:31:28.304329 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:31:28.306903 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:31:28.306909 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:31:28.306924 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:31:28.306954 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:31:28.308768 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:31:28.309251 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:31:28.309261 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:31:28.309266 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:31:28.309380 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:31:28.309679 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:31:28.309684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:31:28.309686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:31:28.310103 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:31:28.310423 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0926 11:31:28.310582 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 11:31:28.310916 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":147,"threads_started":1}
I20260926 11:31:28.313871 11640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c48bf885dae44af886f283909fff17b: No bootstrap required, opened a new log
I20260926 11:31:28.317019 11640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c48bf885dae44af886f283909fff17b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c48bf885dae44af886f283909fff17b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45275 } }
I20260926 11:31:28.317401 11640 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c48bf885dae44af886f283909fff17b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:31:28.317463 11640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c48bf885dae44af886f283909fff17b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c48bf885dae44af886f283909fff17b, State: Initialized, Role: FOLLOWER
I20260926 11:31:28.317936 11640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c48bf885dae44af886f283909fff17b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c48bf885dae44af886f283909fff17b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45275 } }
I20260926 11:31:28.318105 11640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c48bf885dae44af886f283909fff17b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:31:28.318173 11640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c48bf885dae44af886f283909fff17b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:31:28.318269 11640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c48bf885dae44af886f283909fff17b [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:31:28.319242 11640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c48bf885dae44af886f283909fff17b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c48bf885dae44af886f283909fff17b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45275 } }
I20260926 11:31:28.319608 11640 leader_election.cc:302] T 00000000000000000000000000000000 P 2c48bf885dae44af886f283909fff17b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c48bf885dae44af886f283909fff17b; no voters:
I20260926 11:31:28.319964 11640 leader_election.cc:288] T 00000000000000000000000000000000 P 2c48bf885dae44af886f283909fff17b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:31:28.320097 11645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c48bf885dae44af886f283909fff17b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:31:28.320340 11645 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c48bf885dae44af886f283909fff17b [term 1 LEADER]: Becoming Leader. State: Replica: 2c48bf885dae44af886f283909fff17b, State: Running, Role: LEADER
I20260926 11:31:28.320664 11645 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c48bf885dae44af886f283909fff17b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c48bf885dae44af886f283909fff17b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45275 } }
I20260926 11:31:28.321311 11640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c48bf885dae44af886f283909fff17b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:31:28.321997 11645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c48bf885dae44af886f283909fff17b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c48bf885dae44af886f283909fff17b. Latest consensus state: current_term: 1 leader_uuid: "2c48bf885dae44af886f283909fff17b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c48bf885dae44af886f283909fff17b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45275 } } }
I20260926 11:31:28.322118 11645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c48bf885dae44af886f283909fff17b [sys.catalog]: This master's current role is: LEADER
I20260926 11:31:28.322562 11646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c48bf885dae44af886f283909fff17b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c48bf885dae44af886f283909fff17b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c48bf885dae44af886f283909fff17b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45275 } } }
I20260926 11:31:28.322721 11646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c48bf885dae44af886f283909fff17b [sys.catalog]: This master's current role is: LEADER
I20260926 11:31:28.323733 11640 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 11:31:28.324461 11652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:31:28.327819 11652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:31:28.338629 11652 catalog_manager.cc:1347] Generated new cluster ID: 1db291f6118349daaf8d838154d36ce3
I20260926 11:31:28.338730 11652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:31:28.373740 11652 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:31:28.375176 11652 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:31:28.388468 11652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c48bf885dae44af886f283909fff17b: Generated new TSK 0
I20260926 11:31:28.389434 11652 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:31:29.059515 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60815/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45275
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:31:29.178037 11685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:31:29.178310 11685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:31:29.178391 11685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:31:29.182998 11685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:31:29.183084 11685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:31:29.183190 11685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:31:29.188894 11685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:60815/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:31:29.189941 11685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:31:29.190933 11685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:31:29.202021 11690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:31:29.202036 11691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:31:29.202363 11685 server_base.cc:1034] running on GCE node
W20260926 11:31:29.202034 11693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:31:29.203076 11685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:31:29.203929 11685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:31:29.205123 11685 hybrid_clock.cc:648] HybridClock initialized: now 1790422289205092 us; error 48 us; skew 500 ppm
Sep 26 11:31:29 dist-test-slave-mrck krb5kdc[11128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:31:29.208572 11685 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:31:29.209815 11685 webserver.cc:460] Webserver started at http://127.0.98.1:44979/ using document root <none> and password file <none>
I20260926 11:31:29.210605 11685 fs_manager.cc:359] Metadata directory not provided
I20260926 11:31:29.210702 11685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:31:29.210953 11685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 11:31:29 dist-test-slave-mrck krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:31:29.229692 11685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-0/data/instance:
uuid: "1730a1a6fc6e4a98b4825ef7e61f7779"
format_stamp: "Formatted at 2026-09-26 11:31:29 on dist-test-slave-mrck"
server_key: "17fce906cab248c488b948a57b606ed2"
server_key_iv: "9a4044b0584c382f56be0aad13b8b526"
server_key_version: "kuduclusterkey@0"
I20260926 11:31:29.230518 11685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1730a1a6fc6e4a98b4825ef7e61f7779"
format_stamp: "Formatted at 2026-09-26 11:31:29 on dist-test-slave-mrck"
server_key: "17fce906cab248c488b948a57b606ed2"
server_key_iv: "9a4044b0584c382f56be0aad13b8b526"
server_key_version: "kuduclusterkey@0"
I20260926 11:31:29.236619 11685 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260926 11:31:29.259137 11702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:31:29.261235 11685 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260926 11:31:29.261390 11685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "1730a1a6fc6e4a98b4825ef7e61f7779"
format_stamp: "Formatted at 2026-09-26 11:31:29 on dist-test-slave-mrck"
server_key: "17fce906cab248c488b948a57b606ed2"
server_key_iv: "9a4044b0584c382f56be0aad13b8b526"
server_key_version: "kuduclusterkey@0"
I20260926 11:31:29.261541 11685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:31:29.279218 11685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:31:29.283128 11685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:31:29.283339 11685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:31:29.284168 11685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:31:29.285157 11685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:31:29.285265 11685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:31:29.285396 11685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:31:29.285486 11685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:31:29.298310 11685 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44615
I20260926 11:31:29.299863 11685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:31:29.302934 11815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44615 every 8 connection(s)
I20260926 11:31:29.303902 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 11685
I20260926 11:31:29.304051 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:31:29.304450 392 external_mini_cluster.cc:1428] Setting key 3dd6c32ce09862eea293628f514a44f8
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 26 11:31:29 dist-test-slave-mrck krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 11:31:29.340759 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:31:29.308173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37763 (local address 127.0.98.62:45275)
0926 11:31:29.319618 (+ 11445us) server_negotiation.cc:206] Beginning negotiation
0926 11:31:29.319627 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 11:31:29.321263 (+ 1636us) server_negotiation.cc:407] Connection header received
0926 11:31:29.322777 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:31:29.322781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:31:29.322859 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:31:29.322975 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:31:29.324790 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:31:29.325698 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:31:29.328117 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:31:29.328500 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:31:29.328590 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:31:29.335502 (+ 6912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:31:29.335508 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:31:29.335511 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:31:29.335547 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:31:29.337649 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:31:29.338493 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:31:29.338497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:31:29.338499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:31:29.338611 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:31:29.339126 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:31:29.339129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:31:29.339131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:31:29.339389 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:31:29.339482 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 11:31:29.340349 (+ 867us) server_negotiation.cc:299] Negotiation successful
0926 11:31:29.340531 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11331,"thread_start_us":112,"threads_started":1}
I20260926 11:31:29.342859 11820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:31:29.307718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45275 (local address 127.0.98.1:37763)
0926 11:31:29.321002 (+ 13284us) negotiation.cc:107] Waiting for socket to connect
0926 11:31:29.321118 (+ 116us) client_negotiation.cc:174] Beginning negotiation
0926 11:31:29.322473 (+ 1355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:31:29.323350 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:31:29.323362 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:31:29.323791 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:31:29.324563 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:31:29.324587 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:31:29.325924 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:31:29.325927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:31:29.327937 (+ 2010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:31:29.327951 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:31:29.328849 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:31:29.328853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:31:29.329090 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:31:29.329894 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:31:29.329917 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:31:29.335193 (+ 5276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:31:29.337870 (+ 2677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:31:29.337879 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:31:29.337901 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:31:29.338341 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:31:29.338740 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:31:29.338744 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:31:29.338746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:31:29.338991 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:31:29.339558 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:31:29.339566 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:31:29.340025 (+ 459us) client_negotiation.cc:769] Sending connection context
0926 11:31:29.341285 (+ 1260us) client_negotiation.cc:240] Negotiation successful
0926 11:31:29.341689 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12892,"thread_start_us":160,"threads_started":1}
I20260926 11:31:29.344749 11816 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45275
I20260926 11:31:29.345028 11816 heartbeater.cc:464] Registering TS with master...
I20260926 11:31:29.345657 11816 heartbeater.cc:511] Master 127.0.98.62:45275 requested a full tablet report, sending...
I20260926 11:31:29.347681 11604 ts_manager.cc:194] Registered new tserver with Master: 1730a1a6fc6e4a98b4825ef7e61f7779 (127.0.98.1:44615)
I20260926 11:31:29.349365 11604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:31:29.372745 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60815/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45275
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 11:31:29 dist-test-slave-mrck krb5kdc[11128](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790422289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 11:31:29.527062 11824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:31:29.527473 11824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:31:29.527558 11824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:31:29.532841 11824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:31:29.532914 11824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:31:29.533020 11824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:31:29.539978 11824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:60815/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:31:29.541204 11824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:31:29.542318 11824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:31:29.552987 11833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:31:29.553045 11830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:31:29.554312 11824 server_base.cc:1034] running on GCE node
W20260926 11:31:29.554603 11831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:31:29.555069 11824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:31:29.555881 11824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:31:29.557075 11824 hybrid_clock.cc:648] HybridClock initialized: now 1790422289557038 us; error 50 us; skew 500 ppm
Sep 26 11:31:29 dist-test-slave-mrck krb5kdc[11128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:31:29.560508 11824 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:31:29.561446 11824 webserver.cc:460] Webserver started at http://127.0.98.2:45339/ using document root <none> and password file <none>
I20260926 11:31:29.562024 11824 fs_manager.cc:359] Metadata directory not provided
I20260926 11:31:29.562115 11824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:31:29.562330 11824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 11:31:29 dist-test-slave-mrck krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:31:29.575376 11824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-1/data/instance:
uuid: "2d73e8a2ab904e31b10c2a9a3b537f91"
format_stamp: "Formatted at 2026-09-26 11:31:29 on dist-test-slave-mrck"
server_key: "2d39c48f79da202f0fc97efdb63bba23"
server_key_iv: "6072510158cdcdc4c9d813e428b4145c"
server_key_version: "kuduclusterkey@0"
I20260926 11:31:29.576458 11824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2d73e8a2ab904e31b10c2a9a3b537f91"
format_stamp: "Formatted at 2026-09-26 11:31:29 on dist-test-slave-mrck"
server_key: "2d39c48f79da202f0fc97efdb63bba23"
server_key_iv: "6072510158cdcdc4c9d813e428b4145c"
server_key_version: "kuduclusterkey@0"
I20260926 11:31:29.585155 11824 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260926 11:31:29.601148 11840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:31:29.602928 11824 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:31:29.603135 11824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "2d73e8a2ab904e31b10c2a9a3b537f91"
format_stamp: "Formatted at 2026-09-26 11:31:29 on dist-test-slave-mrck"
server_key: "2d39c48f79da202f0fc97efdb63bba23"
server_key_iv: "6072510158cdcdc4c9d813e428b4145c"
server_key_version: "kuduclusterkey@0"
I20260926 11:31:29.603360 11824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:31:29.618386 11824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:31:29.622013 11824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:31:29.622275 11824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:31:29.623020 11824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:31:29.624111 11824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:31:29.624215 11824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:31:29.624312 11824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:31:29.624380 11824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:31:29.635303 11824 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38539
I20260926 11:31:29.637162 11824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:31:29.641826 11953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38539 every 8 connection(s)
I20260926 11:31:29.645870 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 11824
I20260926 11:31:29.646000 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:31:29.646330 392 external_mini_cluster.cc:1428] Setting key 0713eea553f00a0525e354d79c119009
Sep 26 11:31:29 dist-test-slave-mrck krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:31:29.666563 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:31:29.644721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42741 (local address 127.0.98.62:45275)
0926 11:31:29.644870 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 11:31:29.644874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:31:29.645975 (+ 1101us) server_negotiation.cc:407] Connection header received
0926 11:31:29.647557 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:31:29.647561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:31:29.647616 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:31:29.647674 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:31:29.652130 (+ 4456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:31:29.653012 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:31:29.655649 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:31:29.656023 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:31:29.656110 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:31:29.661971 (+ 5861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:31:29.661975 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:31:29.661978 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:31:29.662009 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:31:29.664113 (+ 2104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:31:29.664759 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:31:29.664763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:31:29.664765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:31:29.664842 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:31:29.665203 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:31:29.665207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:31:29.665211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:31:29.665443 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:31:29.665513 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 11:31:29.666241 (+ 728us) server_negotiation.cc:299] Negotiation successful
0926 11:31:29.666386 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 11:31:29.668633 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:31:29.645189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45275 (local address 127.0.98.2:42741)
0926 11:31:29.645782 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0926 11:31:29.645853 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 11:31:29.647262 (+ 1409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:31:29.650077 (+ 2815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:31:29.650088 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:31:29.650496 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:31:29.651241 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:31:29.651263 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:31:29.654020 (+ 2757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:31:29.654024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:31:29.655502 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:31:29.655514 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:31:29.656343 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:31:29.656346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:31:29.656580 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:31:29.659982 (+ 3402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:31:29.660005 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:31:29.661748 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:31:29.664271 (+ 2523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:31:29.664278 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:31:29.664298 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:31:29.664647 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:31:29.664938 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:31:29.664940 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:31:29.664942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:31:29.665107 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:31:29.665590 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:31:29.665596 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:31:29.665952 (+ 356us) client_negotiation.cc:769] Sending connection context
0926 11:31:29.667057 (+ 1105us) client_negotiation.cc:240] Negotiation successful
0926 11:31:29.667438 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":152,"threads_started":1}
I20260926 11:31:29.669974 11954 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45275
I20260926 11:31:29.670228 11954 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 11:31:29.670807 11954 heartbeater.cc:511] Master 127.0.98.62:45275 requested a full tablet report, sending...
I20260926 11:31:29.671898 11604 ts_manager.cc:194] Registered new tserver with Master: 2d73e8a2ab904e31b10c2a9a3b537f91 (127.0.98.2:38539)
I20260926 11:31:29.672583 11604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:31:29.701866 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:60815/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45275
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:31:29.868022 11959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:31:29.868468 11959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:31:29.868590 11959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:31:29.874712 11959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:31:29.874864 11959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:31:29.874996 11959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:31:29.882891 11959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:60815/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:31:29.884671 11959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:31:29.886085 11959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:31:29.899211 11965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:31:29.899787 11964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:31:29.899983 11967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:31:29.900949 11959 server_base.cc:1034] running on GCE node
I20260926 11:31:29.901535 11959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:31:29.902275 11959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
Sep 26 11:31:29 dist-test-slave-mrck krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:31:29.917733 11959 hybrid_clock.cc:648] HybridClock initialized: now 1790422289917606 us; error 165 us; skew 500 ppm
Sep 26 11:31:29 dist-test-slave-mrck krb5kdc[11128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:31:29.921444 11959 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:31:29.922730 11959 webserver.cc:460] Webserver started at http://127.0.98.3:45979/ using document root <none> and password file <none>
I20260926 11:31:29.923437 11959 fs_manager.cc:359] Metadata directory not provided
I20260926 11:31:29.923576 11959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:31:29.923883 11959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 11:31:29 dist-test-slave-mrck krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:31:29.949193 11959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-2/data/instance:
uuid: "ccfeb63117704a4ea627dfe522a736bf"
format_stamp: "Formatted at 2026-09-26 11:31:29 on dist-test-slave-mrck"
server_key: "e3d3b2c37e2ab872eb8f907588eb02d0"
server_key_iv: "1ee66aa852a060903509288d56c598c4"
server_key_version: "kuduclusterkey@0"
I20260926 11:31:29.949990 11959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ccfeb63117704a4ea627dfe522a736bf"
format_stamp: "Formatted at 2026-09-26 11:31:29 on dist-test-slave-mrck"
server_key: "e3d3b2c37e2ab872eb8f907588eb02d0"
server_key_iv: "1ee66aa852a060903509288d56c598c4"
server_key_version: "kuduclusterkey@0"
I20260926 11:31:29.981956 11959 fs_manager.cc:691] Time spent creating directory manager: real 0.032s user 0.005s sys 0.000s
I20260926 11:31:30.016197 11974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:31:30.029377 11959 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.002s sys 0.000s
I20260926 11:31:30.029568 11959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "ccfeb63117704a4ea627dfe522a736bf"
format_stamp: "Formatted at 2026-09-26 11:31:29 on dist-test-slave-mrck"
server_key: "e3d3b2c37e2ab872eb8f907588eb02d0"
server_key_iv: "1ee66aa852a060903509288d56c598c4"
server_key_version: "kuduclusterkey@0"
I20260926 11:31:30.029755 11959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:31:30.063531 11959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:31:30.067353 11959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:31:30.067628 11959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:31:30.068434 11959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:31:30.069505 11959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:31:30.069609 11959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:31:30.069701 11959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:31:30.069761 11959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:31:30.083528 11959 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33823
I20260926 11:31:30.084780 11959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:31:30.085906 12087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33823 every 8 connection(s)
I20260926 11:31:30.086144 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 11959
I20260926 11:31:30.086290 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:31:30.086611 392 external_mini_cluster.cc:1428] Setting key c9f998e954009258c1a5ba5fa2c128fa
Sep 26 11:31:30 dist-test-slave-mrck krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:31:30.105583 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:31:30.090247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56007 (local address 127.0.98.62:45275)
0926 11:31:30.090360 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 11:31:30.090364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:31:30.092228 (+ 1864us) server_negotiation.cc:407] Connection header received
0926 11:31:30.093420 (+ 1192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:31:30.093423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:31:30.093476 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:31:30.093525 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:31:30.095108 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:31:30.095898 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:31:30.097410 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:31:30.097686 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:31:30.097758 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:31:30.101121 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:31:30.101126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:31:30.101130 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:31:30.101164 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:31:30.103117 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:31:30.103795 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:31:30.103798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:31:30.103800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:31:30.103907 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:31:30.104308 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:31:30.104311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:31:30.104313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:31:30.104527 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:31:30.104604 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 11:31:30.105290 (+ 686us) server_negotiation.cc:299] Negotiation successful
0926 11:31:30.105421 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 11:31:30.107774 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:31:30.089236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45275 (local address 127.0.98.3:56007)
0926 11:31:30.092077 (+ 2841us) negotiation.cc:107] Waiting for socket to connect
0926 11:31:30.092135 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 11:31:30.093175 (+ 1040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:31:30.093941 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:31:30.093950 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:31:30.094293 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:31:30.094950 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:31:30.094968 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:31:30.096046 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:31:30.096049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:31:30.097279 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:31:30.097290 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:31:30.097966 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:31:30.097969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:31:30.098156 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:31:30.098842 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:31:30.098862 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:31:30.100901 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:31:30.103281 (+ 2380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:31:30.103289 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:31:30.103307 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:31:30.103646 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:31:30.104019 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:31:30.104022 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:31:30.104024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:31:30.104206 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:31:30.104670 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:31:30.104676 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:31:30.105016 (+ 340us) client_negotiation.cc:769] Sending connection context
0926 11:31:30.106344 (+ 1328us) client_negotiation.cc:240] Negotiation successful
0926 11:31:30.106696 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2617,"thread_start_us":123,"threads_started":1}
I20260926 11:31:30.109241 12088 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45275
I20260926 11:31:30.109476 12088 heartbeater.cc:464] Registering TS with master...
I20260926 11:31:30.110029 12088 heartbeater.cc:511] Master 127.0.98.62:45275 requested a full tablet report, sending...
I20260926 11:31:30.111194 11604 ts_manager.cc:194] Registered new tserver with Master: ccfeb63117704a4ea627dfe522a736bf (127.0.98.3:33823)
I20260926 11:31:30.111963 11604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56007
I20260926 11:31:30.112550 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:31:30.125339 11819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:31:30.114790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56840 (local address 127.0.98.62:45275)
0926 11:31:30.114901 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 11:31:30.114905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:31:30.115474 (+ 569us) server_negotiation.cc:407] Connection header received
0926 11:31:30.115639 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:31:30.115642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:31:30.115684 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:31:30.115759 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:31:30.116690 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:31:30.117470 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:31:30.118670 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:31:30.119014 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:31:30.119082 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:31:30.121111 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:31:30.121115 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:31:30.121118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:31:30.121143 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:31:30.122931 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:31:30.123442 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:31:30.123446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:31:30.123448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:31:30.123527 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:31:30.123848 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:31:30.123851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:31:30.123853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:31:30.124051 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:31:30.124148 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 11:31:30.125061 (+ 913us) server_negotiation.cc:299] Negotiation successful
0926 11:31:30.125177 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 11:31:30.129113 11601 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:31:30.133482 11601 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:31:30.162979 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:31:30.144742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38539 (local address 127.0.0.1:56166)
0926 11:31:30.149216 (+ 4474us) negotiation.cc:107] Waiting for socket to connect
0926 11:31:30.149267 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 11:31:30.149395 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:31:30.153565 (+ 4170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:31:30.153568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:31:30.153583 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:31:30.153659 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:31:30.153664 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:31:30.157269 (+ 3605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:31:30.157272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:31:30.160314 (+ 3042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:31:30.160323 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:31:30.162427 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:31:30.162431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:31:30.162536 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:31:30.162572 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 11:31:30.162679 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 11:31:30.162816 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4298,"mutex_wait_us":1691,"thread_start_us":101,"threads_started":1}
I20260926 11:31:30.164753 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:31:30.148137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58516 (local address 127.0.98.1:44615)
0926 11:31:30.155937 (+ 7800us) server_negotiation.cc:206] Beginning negotiation
0926 11:31:30.155944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:31:30.155970 (+ 26us) server_negotiation.cc:407] Connection header received
0926 11:31:30.156043 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:31:30.156049 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:31:30.156253 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:31:30.156413 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:31:30.156771 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:31:30.157840 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:31:30.159342 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:31:30.160173 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:31:30.164297 (+ 4124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:31:30.164454 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 11:31:30.164513 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 11:31:30.164621 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7686,"thread_start_us":73,"threads_started":1}
I20260926 11:31:30.164170 12105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:31:30.147993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44615 (local address 127.0.0.1:58516)
0926 11:31:30.151840 (+ 3847us) negotiation.cc:107] Waiting for socket to connect
0926 11:31:30.151857 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:31:30.152004 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:31:30.156543 (+ 4539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:31:30.156546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:31:30.156562 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:31:30.156646 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:31:30.156652 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:31:30.157964 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:31:30.157968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:31:30.159178 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:31:30.159190 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:31:30.163784 (+ 4594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:31:30.163789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:31:30.163884 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:31:30.163908 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 11:31:30.163961 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 11:31:30.164034 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3748,"mutex_wait_us":1,"thread_start_us":90,"threads_started":1}
I20260926 11:31:30.165477 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:31:30.146929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33823 (local address 127.0.0.1:38676)
0926 11:31:30.149807 (+ 2878us) negotiation.cc:107] Waiting for socket to connect
0926 11:31:30.149819 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 11:31:30.149897 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:31:30.152842 (+ 2945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:31:30.152848 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:31:30.152871 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:31:30.153004 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:31:30.153010 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:31:30.155078 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:31:30.155081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:31:30.160645 (+ 5564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:31:30.160652 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:31:30.165094 (+ 4442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:31:30.165098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:31:30.165200 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:31:30.165226 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 11:31:30.165284 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 11:31:30.165357 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2815,"thread_start_us":73,"threads_started":1}
I20260926 11:31:30.165951 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:31:30.147283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38676 (local address 127.0.98.3:33823)
0926 11:31:30.149676 (+ 2393us) server_negotiation.cc:206] Beginning negotiation
0926 11:31:30.149687 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 11:31:30.149975 (+ 288us) server_negotiation.cc:407] Connection header received
0926 11:31:30.150057 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:31:30.150062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:31:30.150269 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:31:30.150416 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:31:30.153798 (+ 3382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:31:30.154952 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:31:30.160825 (+ 5873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:31:30.164759 (+ 3934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:31:30.164888 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:31:30.165037 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 11:31:30.165709 (+ 672us) server_negotiation.cc:299] Negotiation successful
0926 11:31:30.165819 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2291,"thread_start_us":93,"threads_started":1}
I20260926 11:31:30.166992 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:31:30.145641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56166 (local address 127.0.98.2:38539)
0926 11:31:30.150677 (+ 5036us) server_negotiation.cc:206] Beginning negotiation
0926 11:31:30.150690 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 11:31:30.150711 (+ 21us) server_negotiation.cc:407] Connection header received
0926 11:31:30.150776 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:31:30.150792 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:31:30.153296 (+ 2504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:31:30.153436 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:31:30.156022 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:31:30.157140 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:31:30.161333 (+ 4193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:31:30.162113 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:31:30.162216 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:31:30.162367 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 11:31:30.166748 (+ 4381us) server_negotiation.cc:299] Negotiation successful
0926 11:31:30.166862 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4935,"thread_start_us":95,"threads_started":1}
I20260926 11:31:30.168710 11749 tablet_service.cc:1467] Processing CreateTablet for tablet 52d36ecb6dfb45dcb00fcbd62f7aaa45 (DEFAULT_TABLE table=test-table [id=8ec3451611ef495483506b85c23b7ca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:31:30.168951 12022 tablet_service.cc:1467] Processing CreateTablet for tablet 52d36ecb6dfb45dcb00fcbd62f7aaa45 (DEFAULT_TABLE table=test-table [id=8ec3451611ef495483506b85c23b7ca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:31:30.169867 11749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52d36ecb6dfb45dcb00fcbd62f7aaa45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:31:30.169921 12022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52d36ecb6dfb45dcb00fcbd62f7aaa45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:31:30.169862 11880 tablet_service.cc:1467] Processing CreateTablet for tablet 52d36ecb6dfb45dcb00fcbd62f7aaa45 (DEFAULT_TABLE table=test-table [id=8ec3451611ef495483506b85c23b7ca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:31:30.170699 11880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52d36ecb6dfb45dcb00fcbd62f7aaa45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:31:30.179560 12107 tablet_bootstrap.cc:492] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 1730a1a6fc6e4a98b4825ef7e61f7779: Bootstrap starting.
I20260926 11:31:30.180028 12108 tablet_bootstrap.cc:492] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf: Bootstrap starting.
I20260926 11:31:30.182478 12107 tablet_bootstrap.cc:654] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 1730a1a6fc6e4a98b4825ef7e61f7779: Neither blocks nor log segments found. Creating new log.
I20260926 11:31:30.183543 12108 tablet_bootstrap.cc:654] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf: Neither blocks nor log segments found. Creating new log.
I20260926 11:31:30.183511 12107 log.cc:824] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 1730a1a6fc6e4a98b4825ef7e61f7779: Log is configured to *not* fsync() on all Append() calls
I20260926 11:31:30.184360 12108 log.cc:824] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf: Log is configured to *not* fsync() on all Append() calls
I20260926 11:31:30.185009 12109 tablet_bootstrap.cc:492] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 2d73e8a2ab904e31b10c2a9a3b537f91: Bootstrap starting.
I20260926 11:31:30.185721 12107 tablet_bootstrap.cc:492] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 1730a1a6fc6e4a98b4825ef7e61f7779: No bootstrap required, opened a new log
I20260926 11:31:30.185912 12107 ts_tablet_manager.cc:1397] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 1730a1a6fc6e4a98b4825ef7e61f7779: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 11:31:30.187624 12109 tablet_bootstrap.cc:654] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 2d73e8a2ab904e31b10c2a9a3b537f91: Neither blocks nor log segments found. Creating new log.
I20260926 11:31:30.189481 12109 log.cc:824] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 2d73e8a2ab904e31b10c2a9a3b537f91: Log is configured to *not* fsync() on all Append() calls
I20260926 11:31:30.191663 12108 tablet_bootstrap.cc:492] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf: No bootstrap required, opened a new log
I20260926 11:31:30.191484 12107 raft_consensus.cc:348] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 1730a1a6fc6e4a98b4825ef7e61f7779 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d73e8a2ab904e31b10c2a9a3b537f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38539 } } peers { permanent_uuid: "ccfeb63117704a4ea627dfe522a736bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33823 } } peers { permanent_uuid: "1730a1a6fc6e4a98b4825ef7e61f7779" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44615 } }
I20260926 11:31:30.191877 12107 raft_consensus.cc:374] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 1730a1a6fc6e4a98b4825ef7e61f7779 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:31:30.191956 12108 ts_tablet_manager.cc:1397] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260926 11:31:30.191985 12107 raft_consensus.cc:729] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 1730a1a6fc6e4a98b4825ef7e61f7779 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1730a1a6fc6e4a98b4825ef7e61f7779, State: Initialized, Role: FOLLOWER
I20260926 11:31:30.192595 12107 consensus_queue.cc:260] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 1730a1a6fc6e4a98b4825ef7e61f7779 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d73e8a2ab904e31b10c2a9a3b537f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38539 } } peers { permanent_uuid: "ccfeb63117704a4ea627dfe522a736bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33823 } } peers { permanent_uuid: "1730a1a6fc6e4a98b4825ef7e61f7779" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44615 } }
I20260926 11:31:30.193703 12107 ts_tablet_manager.cc:1428] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 1730a1a6fc6e4a98b4825ef7e61f7779: Time spent starting tablet: real 0.007s user 0.002s sys 0.006s
I20260926 11:31:30.194085 11816 heartbeater.cc:503] Master 127.0.98.62:45275 was elected leader, sending a full tablet report...
I20260926 11:31:30.195762 12108 raft_consensus.cc:348] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d73e8a2ab904e31b10c2a9a3b537f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38539 } } peers { permanent_uuid: "ccfeb63117704a4ea627dfe522a736bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33823 } } peers { permanent_uuid: "1730a1a6fc6e4a98b4825ef7e61f7779" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44615 } }
I20260926 11:31:30.196007 12108 raft_consensus.cc:374] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:31:30.196054 12108 raft_consensus.cc:729] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccfeb63117704a4ea627dfe522a736bf, State: Initialized, Role: FOLLOWER
I20260926 11:31:30.196527 12108 consensus_queue.cc:260] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d73e8a2ab904e31b10c2a9a3b537f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38539 } } peers { permanent_uuid: "ccfeb63117704a4ea627dfe522a736bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33823 } } peers { permanent_uuid: "1730a1a6fc6e4a98b4825ef7e61f7779" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44615 } }
I20260926 11:31:30.197508 12108 ts_tablet_manager.cc:1428] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 11:31:30.197887 12088 heartbeater.cc:503] Master 127.0.98.62:45275 was elected leader, sending a full tablet report...
I20260926 11:31:30.202205 12109 tablet_bootstrap.cc:492] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 2d73e8a2ab904e31b10c2a9a3b537f91: No bootstrap required, opened a new log
I20260926 11:31:30.202447 12109 ts_tablet_manager.cc:1397] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 2d73e8a2ab904e31b10c2a9a3b537f91: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20260926 11:31:30.205751 12109 raft_consensus.cc:348] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 2d73e8a2ab904e31b10c2a9a3b537f91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d73e8a2ab904e31b10c2a9a3b537f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38539 } } peers { permanent_uuid: "ccfeb63117704a4ea627dfe522a736bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33823 } } peers { permanent_uuid: "1730a1a6fc6e4a98b4825ef7e61f7779" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44615 } }
I20260926 11:31:30.205997 12109 raft_consensus.cc:374] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 2d73e8a2ab904e31b10c2a9a3b537f91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:31:30.206099 12109 raft_consensus.cc:729] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 2d73e8a2ab904e31b10c2a9a3b537f91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d73e8a2ab904e31b10c2a9a3b537f91, State: Initialized, Role: FOLLOWER
I20260926 11:31:30.206648 12109 consensus_queue.cc:260] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 2d73e8a2ab904e31b10c2a9a3b537f91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d73e8a2ab904e31b10c2a9a3b537f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38539 } } peers { permanent_uuid: "ccfeb63117704a4ea627dfe522a736bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33823 } } peers { permanent_uuid: "1730a1a6fc6e4a98b4825ef7e61f7779" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44615 } }
I20260926 11:31:30.207793 12109 ts_tablet_manager.cc:1428] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 2d73e8a2ab904e31b10c2a9a3b537f91: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260926 11:31:30.208184 11954 heartbeater.cc:503] Master 127.0.98.62:45275 was elected leader, sending a full tablet report...
Sep 26 11:31:30 dist-test-slave-mrck krb5kdc[11128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260926 11:31:30.306509 11817 tablet.cc:2369] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 1730a1a6fc6e4a98b4825ef7e61f7779: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 11:31:30.337714 12089 tablet.cc:2369] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 11:31:30.393652 11955 tablet.cc:2369] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 2d73e8a2ab904e31b10c2a9a3b537f91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:31:30.464556 12114 raft_consensus.cc:482] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:31:30.464841 12114 raft_consensus.cc:504] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d73e8a2ab904e31b10c2a9a3b537f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38539 } } peers { permanent_uuid: "ccfeb63117704a4ea627dfe522a736bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33823 } } peers { permanent_uuid: "1730a1a6fc6e4a98b4825ef7e61f7779" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44615 } }
I20260926 11:31:30.466274 12114 leader_election.cc:288] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d73e8a2ab904e31b10c2a9a3b537f91 (127.0.98.2:38539), 1730a1a6fc6e4a98b4825ef7e61f7779 (127.0.98.1:44615)
I20260926 11:31:30.471374 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:31:30.466516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38539 (local address 127.0.98.3:41307)
0926 11:31:30.466640 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 11:31:30.466657 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:31:30.466773 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:31:30.467468 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:31:30.467472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:31:30.467494 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:31:30.467605 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:31:30.467611 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:31:30.468636 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:31:30.468640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:31:30.469926 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:31:30.469937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:31:30.470944 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:31:30.470947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:31:30.471043 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:31:30.471070 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 11:31:30.471144 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 11:31:30.471214 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":46}
I20260926 11:31:30.471534 12106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:31:30.467001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48081 (local address 127.0.98.1:44615)
0926 11:31:30.467152 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 11:31:30.467157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:31:30.467170 (+ 13us) server_negotiation.cc:407] Connection header received
0926 11:31:30.467306 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:31:30.467310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:31:30.467362 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:31:30.467412 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:31:30.467886 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:31:30.468640 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:31:30.470188 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:31:30.470823 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:31:30.470903 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:31:30.470966 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 11:31:30.471326 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 11:31:30.471410 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 11:31:30.471918 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:31:30.466607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44615 (local address 127.0.98.3:48081)
0926 11:31:30.467069 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0926 11:31:30.467081 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 11:31:30.467165 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:31:30.467571 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:31:30.467574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:31:30.467613 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:31:30.467722 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:31:30.467754 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:31:30.468789 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:31:30.468792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:31:30.470042 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:31:30.470053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:31:30.471061 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:31:30.471064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:31:30.471164 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:31:30.471188 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 11:31:30.471667 (+ 479us) client_negotiation.cc:240] Negotiation successful
0926 11:31:30.471759 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"mutex_wait_us":1,"thread_start_us":115,"threads_started":1}
I20260926 11:31:30.472146 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:31:30.466786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41307 (local address 127.0.98.2:38539)
0926 11:31:30.466903 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 11:31:30.466909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:31:30.466926 (+ 17us) server_negotiation.cc:407] Connection header received
0926 11:31:30.466982 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:31:30.466985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:31:30.467036 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:31:30.467132 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:31:30.467764 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:31:30.468509 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:31:30.470091 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:31:30.470720 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:31:30.470819 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:31:30.470885 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 11:31:30.471957 (+ 1072us) server_negotiation.cc:299] Negotiation successful
0926 11:31:30.472035 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 11:31:30.472543 11759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52d36ecb6dfb45dcb00fcbd62f7aaa45" candidate_uuid: "ccfeb63117704a4ea627dfe522a736bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1730a1a6fc6e4a98b4825ef7e61f7779" is_pre_election: true
I20260926 11:31:30.472708 11908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52d36ecb6dfb45dcb00fcbd62f7aaa45" candidate_uuid: "ccfeb63117704a4ea627dfe522a736bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d73e8a2ab904e31b10c2a9a3b537f91" is_pre_election: true
I20260926 11:31:30.472898 11759 raft_consensus.cc:2454] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 1730a1a6fc6e4a98b4825ef7e61f7779 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccfeb63117704a4ea627dfe522a736bf in term 0.
I20260926 11:31:30.472996 11908 raft_consensus.cc:2454] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 2d73e8a2ab904e31b10c2a9a3b537f91 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccfeb63117704a4ea627dfe522a736bf in term 0.
I20260926 11:31:30.473421 11977 leader_election.cc:302] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1730a1a6fc6e4a98b4825ef7e61f7779, ccfeb63117704a4ea627dfe522a736bf; no voters:
I20260926 11:31:30.473762 12114 raft_consensus.cc:2781] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:31:30.473846 12114 raft_consensus.cc:482] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:31:30.473888 12114 raft_consensus.cc:3037] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:31:30.475232 12114 raft_consensus.cc:504] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d73e8a2ab904e31b10c2a9a3b537f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38539 } } peers { permanent_uuid: "ccfeb63117704a4ea627dfe522a736bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33823 } } peers { permanent_uuid: "1730a1a6fc6e4a98b4825ef7e61f7779" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44615 } }
I20260926 11:31:30.475636 12114 leader_election.cc:288] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf [CANDIDATE]: Term 1 election: Requested vote from peers 2d73e8a2ab904e31b10c2a9a3b537f91 (127.0.98.2:38539), 1730a1a6fc6e4a98b4825ef7e61f7779 (127.0.98.1:44615)
I20260926 11:31:30.476006 11908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52d36ecb6dfb45dcb00fcbd62f7aaa45" candidate_uuid: "ccfeb63117704a4ea627dfe522a736bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d73e8a2ab904e31b10c2a9a3b537f91"
I20260926 11:31:30.476131 11908 raft_consensus.cc:3037] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 2d73e8a2ab904e31b10c2a9a3b537f91 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:31:30.476229 11759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52d36ecb6dfb45dcb00fcbd62f7aaa45" candidate_uuid: "ccfeb63117704a4ea627dfe522a736bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1730a1a6fc6e4a98b4825ef7e61f7779"
I20260926 11:31:30.476366 11759 raft_consensus.cc:3037] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 1730a1a6fc6e4a98b4825ef7e61f7779 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:31:30.477459 11908 raft_consensus.cc:2454] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 2d73e8a2ab904e31b10c2a9a3b537f91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccfeb63117704a4ea627dfe522a736bf in term 1.
I20260926 11:31:30.477563 11759 raft_consensus.cc:2454] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 1730a1a6fc6e4a98b4825ef7e61f7779 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ccfeb63117704a4ea627dfe522a736bf in term 1.
I20260926 11:31:30.477823 11976 leader_election.cc:302] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d73e8a2ab904e31b10c2a9a3b537f91, ccfeb63117704a4ea627dfe522a736bf; no voters:
I20260926 11:31:30.477988 12114 raft_consensus.cc:2781] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:31:30.478252 12114 raft_consensus.cc:686] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf [term 1 LEADER]: Becoming Leader. State: Replica: ccfeb63117704a4ea627dfe522a736bf, State: Running, Role: LEADER
I20260926 11:31:30.478631 12114 consensus_queue.cc:237] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d73e8a2ab904e31b10c2a9a3b537f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38539 } } peers { permanent_uuid: "ccfeb63117704a4ea627dfe522a736bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33823 } } peers { permanent_uuid: "1730a1a6fc6e4a98b4825ef7e61f7779" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44615 } }
I20260926 11:31:30.482263 11604 catalog_manager.cc:5495] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf reported cstate change: term changed from 0 to 1, leader changed from <none> to ccfeb63117704a4ea627dfe522a736bf (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ccfeb63117704a4ea627dfe522a736bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d73e8a2ab904e31b10c2a9a3b537f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38539 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccfeb63117704a4ea627dfe522a736bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33823 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1730a1a6fc6e4a98b4825ef7e61f7779" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44615 } health_report { overall_health: UNKNOWN } } }
I20260926 11:31:30.521832 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:31:30.517699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38688 (local address 127.0.98.3:33823)
0926 11:31:30.517849 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 11:31:30.517854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:31:30.517871 (+ 17us) server_negotiation.cc:407] Connection header received
0926 11:31:30.517960 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:31:30.517963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:31:30.518031 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:31:30.518150 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:31:30.518389 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:31:30.519157 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:31:30.520057 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:31:30.520394 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:31:30.520498 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:31:30.520759 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:31:30.521332 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:31:30.521462 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 11:31:30.521609 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 11:31:30.521670 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 11:31:30.528223 11759 raft_consensus.cc:1261] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 1730a1a6fc6e4a98b4825ef7e61f7779 [term 1 FOLLOWER]: Refusing update from remote peer ccfeb63117704a4ea627dfe522a736bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:31:30.528223 11908 raft_consensus.cc:1261] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 2d73e8a2ab904e31b10c2a9a3b537f91 [term 1 FOLLOWER]: Refusing update from remote peer ccfeb63117704a4ea627dfe522a736bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:31:30.529018 12114 consensus_queue.cc:1035] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1730a1a6fc6e4a98b4825ef7e61f7779" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:31:30.529210 12119 consensus_queue.cc:1035] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d73e8a2ab904e31b10c2a9a3b537f91" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38539 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:31:30.540876 12130 mvcc.cc:205] Tried to move back new op lower bound from 7333569701996126208 to 7333569701803712512. Current Snapshot: MvccSnapshot[applied={T|T < 7333569701996126208}]
I20260926 11:31:30.549585 11604 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:31:30.549770 11604 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:31:30.552659 11604 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c48bf885dae44af886f283909fff17b: Sending DeleteTablet for 3 replicas of tablet 52d36ecb6dfb45dcb00fcbd62f7aaa45
I20260926 11:31:30.553498 11880 tablet_service.cc:1514] Processing DeleteTablet for tablet 52d36ecb6dfb45dcb00fcbd62f7aaa45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:31:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56166
I20260926 11:31:30.553617 12022 tablet_service.cc:1514] Processing DeleteTablet for tablet 52d36ecb6dfb45dcb00fcbd62f7aaa45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:31:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38676
I20260926 11:31:30.554020 11749 tablet_service.cc:1514] Processing DeleteTablet for tablet 52d36ecb6dfb45dcb00fcbd62f7aaa45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:31:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58516
I20260926 11:31:30.554286 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 11685
I20260926 11:31:30.555166 12139 tablet_replica.cc:330] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 2d73e8a2ab904e31b10c2a9a3b537f91: stopping tablet replica
I20260926 11:31:30.555540 12139 raft_consensus.cc:2229] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 2d73e8a2ab904e31b10c2a9a3b537f91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:31:30.555975 12139 raft_consensus.cc:2258] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 2d73e8a2ab904e31b10c2a9a3b537f91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:31:30.556370 12140 tablet_replica.cc:330] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf: stopping tablet replica
I20260926 11:31:30.556686 12140 raft_consensus.cc:2229] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:31:30.557111 12140 raft_consensus.cc:2258] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:31:30.558609 12140 ts_tablet_manager.cc:1905] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:31:30.559861 12139 ts_tablet_manager.cc:1905] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P 2d73e8a2ab904e31b10c2a9a3b537f91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:31:30.562098 12140 ts_tablet_manager.cc:1918] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:31:30.562194 12140 log.cc:1196] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790422180210734-392-0/minicluster-data/ts-2/wal/wals/52d36ecb6dfb45dcb00fcbd62f7aaa45
I20260926 11:31:30.562525 12140 ts_tablet_manager.cc:1939] T 52d36ecb6dfb45dcb00fcbd62f7aaa45 P ccfeb63117704a4ea627dfe522a736bf: Deleting consensus metadata
I20260926 11:31:30.563721 11591 catalog_manager.cc:4841] TS ccfeb63117704a4ea627dfe522a736bf (127.0.98.3:33823): tablet 52d36ecb6dfb45dcb00fcbd62f7aaa45 (table test-table [id=8ec3451611ef495483506b85c23b7ca5]) successfully deleted
W20260926 11:31:30.564105 11590 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44615 (error 108)
I20260926 11:31:30.565510 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 11824
I20260926 11:31:30.565553 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:31:30.565282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44615 (local address 127.0.0.1:58532)
0926 11:31:30.565396 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 11:31:30.565466 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 11:31:30.565704 11590 catalog_manager.cc:4584] TS 1730a1a6fc6e4a98b4825ef7e61f7779 (127.0.98.1:44615): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 52d36ecb6dfb45dcb00fcbd62f7aaa45: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44615: connect: Connection refused (error 111)
I20260926 11:31:30.572949 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:31:30.572598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38539 (local address 127.0.0.1:56182)
0926 11:31:30.572779 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 11:31:30.572843 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38539: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260926 11:31:30.573421 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 11959
I20260926 11:31:30.578677 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 11572
I20260926 11:31:30.583907 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 11:31:30.922537 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 11:31:30.922703 392 mini_ranger.cc:66] Stopping Ranger...
I20260926 11:31:31.269294 392 mini_ranger.cc:68] Stopped Ranger
2026-09-26 11:31:31.269 UTC [11158] LOG: received smart shutdown request
2026-09-26 11:31:31.272 UTC [11158] LOG: background worker "logical replication launcher" (PID 11167) exited with exit code 1
2026-09-26 11:31:31.273 UTC [11162] LOG: shutting down
2026-09-26 11:31:31.440 UTC [11158] LOG: database system is shut down
2026-09-26T11:31:31Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46094 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:31:31 dist-test-slave-mrck krb5kdc[12151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:31:31 dist-test-slave-mrck krb5kdc[12151](info): set up 2 sockets
Sep 26 11:31:31 dist-test-slave-mrck krb5kdc[12151](info): commencing operation
krb5kdc: starting...
W20260926 11:31:32.591848 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:31:32 dist-test-slave-mrck krb5kdc[12151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:31:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:31:32Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260926 11:31:32.764643 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskF2HClO/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/postgres -l logfile start
2026-09-26 11:31:35.186 UTC [12181] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 11:31:35.186 UTC [12181] LOG: listening on IPv4 address "127.0.98.20", port 35135
2026-09-26 11:31:35.189 UTC [12181] LOG: listening on Unix socket "/tmp/.s.PGSQL.35135"
2026-09-26 11:31:35.223 UTC [12184] LOG: database system was shut down at 2026-09-26 11:31:33 UTC
2026-09-26 11:31:35.226 UTC [12181] LOG: database system is ready to accept connections
I20260926 11:31:36.159915 392 mini_postgres.cc:96] Postgres bound to 35135
2026-09-26 11:31:36.165 UTC [12192] FATAL: database "slave" does not exist
127.0.98.20:35135 - accepting connections
I20260926 11:31:36.166477 392 mini_ranger.cc:161] Starting Ranger...
I20260926 11:31:36.181159 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 11:31:36.657668 392 mini_ranger.cc:87] Created ranger Postgres database
I20260926 11:31:36.657773 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ranger-admin
2026-09-26 11:31:36,938 [I] DB FLAVOR :POSTGRES
2026-09-26 11:31:36,939 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 11:31:36,939 [I] Checking connection..
2026-09-26 11:31:36,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 11:31:37,202 [I] Checking connection passed.
2026-09-26 11:31:37,203 [I] --------- Verifying version history table ---------
2026-09-26 11:31:37,203 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 11:31:37,449 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 11:31:37,449 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 11:31:37,695 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 11:31:37,695 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 11:31:37,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35135/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 11:31:37.941 UTC [12256] WARNING: there is no transaction in progress
2026-09-26 11:31:37,947 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 11:31:37,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 11:31:38,192 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 11:31:38,193 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 11:31:38,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 11:31:38,456 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 11:31:38,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-mrck.c.gcp-upstream.internal','N') ;"
2026-09-26 11:31:38,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 11:31:39,220 [I] Table x_portal_user does not exist in database ranger
2026-09-26 11:31:39,221 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 11:31:39,463 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 11:31:39,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 11:31:39,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 11:31:39,980 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 11:31:39,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35135/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 11:31:40.967 UTC [12364] WARNING: there is no transaction in progress
2026-09-26 11:31:40.983 UTC [12364] WARNING: there is no transaction in progress
2026-09-26 11:31:40.999 UTC [12364] WARNING: there is no transaction in progress
2026-09-26 11:31:41.014 UTC [12364] WARNING: there is no transaction in progress
2026-09-26 11:31:41.029 UTC [12364] WARNING: there is no transaction in progress
2026-09-26 11:31:41.046 UTC [12364] WARNING: there is no transaction in progress
2026-09-26 11:31:41.101 UTC [12364] WARNING: there is no transaction in progress
2026-09-26 11:31:41.108 UTC [12364] WARNING: there is no transaction in progress
2026-09-26 11:31:41.122 UTC [12364] WARNING: there is no transaction in progress
2026-09-26 11:31:41.132 UTC [12364] WARNING: there is no transaction in progress
2026-09-26 11:31:41.142 UTC [12364] WARNING: there is no transaction in progress
2026-09-26 11:31:41.155 UTC [12364] WARNING: there is no transaction in progress
2026-09-26 11:31:41.164 UTC [12364] WARNING: there is no transaction in progress
2026-09-26 11:31:41.171 UTC [12364] WARNING: there is no transaction in progress
2026-09-26 11:31:41.178 UTC [12364] WARNING: there is no transaction in progress
2026-09-26 11:31:41.185 UTC [12364] WARNING: there is no transaction in progress
2026-09-26 11:31:41,827 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 11:31:41,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 11:31:42,051 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 11:31:42,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 11:31:42,317 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 11:31:42,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 11:31:42,570 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 11:31:42,570 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 11:31:42,830 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 11:31:43,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-mrck.c.gcp-upstream.internal';"
2026-09-26 11:31:43,297 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 11:31:43,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 11:31:43,541 [I] DB_PATCHES have already been applied
I20260926 11:31:43.546761 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskF2HClO/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 11:31:43.546820 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260926 11:31:43.550071 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:37879
Sep 26, 2026 11:31:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 11:31:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 11:31:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 11:31:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 11:31:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 11:31:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 11:31:44 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 11:31:44.659101 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.493s user 0.002s sys 0.006s
Sep 26, 2026 11:32:05 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 11:32:05 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 11:32:05 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 11:32:05 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 11:32:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 11:32:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 11:32:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 11:32:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 11:32:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 11:32:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 11:32:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 11:32:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 11:32:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 11:32:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 11:32:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 11:32:05 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 11:32:05 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 11:32:06 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 11:32:06.692325 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 11:32:06 dist-test-slave-mrck krb5kdc[12151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422326, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 11:32:06.778697 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 11:32:06.788625 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 11:32:07.702126 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 11:32:07.702219 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ranger-kms
I20260926 11:32:07.702239 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:35135
2026-09-26 11:32:07,872 [I] DB FLAVOR :POSTGRES
2026-09-26 11:32:07,872 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 11:32:07,872 [I] Checking connection
2026-09-26 11:32:07,872 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35135/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 11:32:08,120 [I] connection success
2026-09-26 11:32:08,120 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 11:32:08,120 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 11:32:08,120 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35135/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 11:32:08,350 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 11:32:08,350 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 11:32:08,350 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 11:32:08,351 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35135/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 11:32:08,640 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 11:32:09.159078 392 mini_ranger_kms.cc:310] Created kms service
I20260926 11:32:09.342190 392 mini_ranger_kms.cc:326] Created kudu user
I20260926 11:32:09.390823 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 11:32:09 dist-test-slave-mrck krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422326, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:32:09.808415 392 mini_ranger_kms.cc:384] Added ranger policy
I20260926 11:32:09.808584 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskF2HClO/build/debug/bin/postgresql.jar:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskF2HClO/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 11:32:09.808616 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260926 11:32:09.812420 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12544
I20260926 11:32:09.812508 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:58293
Sep 26, 2026 11:32:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 11:32:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 11:32:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 11:32:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 11:32:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 11:32:11 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 11:32:12.097805 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.319s user 0.006s sys 0.149s
I20260926 11:32:12.097942 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 11:32:12.098014 392 mini_ranger_kms.cc:401] 127.0.98.20:58293/kms/v1/keys
Sep 26 11:32:12 dist-test-slave-mrck krb5kdc[12151](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790422332, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 11:32:12 dist-test-slave-mrck krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422332, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 11:32:13 dist-test-slave-mrck krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422332, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 11:32:15 dist-test-slave-mrck krb5kdc[12151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:32:15.575418 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33801
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33801
--ranger_config_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58293/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:32:15.705603 12595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:32:15.705857 12595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:32:15.705894 12595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:32:15.710005 12595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:32:15.710069 12595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:32:15.710088 12595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:32:15.710104 12595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:32:15.710163 12595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:32:15.715710 12595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:58293/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33801
--ranger_config_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:32:15.716823 12595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:32:15.717803 12595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:32:15.726686 12600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:15.726739 12603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:15.726930 12595 server_base.cc:1034] running on GCE node
W20260926 11:32:15.726739 12601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:15.727602 12595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:32:15.728729 12595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:32:15.729923 12595 hybrid_clock.cc:648] HybridClock initialized: now 1790422335729893 us; error 48 us; skew 500 ppm
Sep 26 11:32:15 dist-test-slave-mrck krb5kdc[12151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:32:15.736848 12595 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:32:15.737838 12595 webserver.cc:460] Webserver started at http://127.0.98.62:37153/ using document root <none> and password file <none>
I20260926 11:32:15.738379 12595 fs_manager.cc:359] Metadata directory not provided
I20260926 11:32:15.738435 12595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:32:15.738653 12595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 11:32:15 dist-test-slave-mrck krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:32:15.798967 12595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "272dc9d37b684ad7aa3fd9fca08ea68a"
format_stamp: "Formatted at 2026-09-26 11:32:15 on dist-test-slave-mrck"
server_key: "ce0139291c9469abee62268fc83f68c7"
server_key_iv: "f3b7ab46c6deca421e526062af330425"
server_key_version: "kuduclusterkey@0"
I20260926 11:32:15.799607 12595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "272dc9d37b684ad7aa3fd9fca08ea68a"
format_stamp: "Formatted at 2026-09-26 11:32:15 on dist-test-slave-mrck"
server_key: "ce0139291c9469abee62268fc83f68c7"
server_key_iv: "f3b7ab46c6deca421e526062af330425"
server_key_version: "kuduclusterkey@0"
I20260926 11:32:15.803917 12595 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 11:32:15.821259 12610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:15.822758 12595 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 11:32:15.822911 12595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "272dc9d37b684ad7aa3fd9fca08ea68a"
format_stamp: "Formatted at 2026-09-26 11:32:15 on dist-test-slave-mrck"
server_key: "ce0139291c9469abee62268fc83f68c7"
server_key_iv: "f3b7ab46c6deca421e526062af330425"
server_key_version: "kuduclusterkey@0"
I20260926 11:32:15.823166 12595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:32:15.853188 12595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:32:15.856535 12595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:32:15.856761 12595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:32:15.864989 12595 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33801
I20260926 11:32:15.864995 12662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33801 every 8 connection(s)
I20260926 11:32:15.866080 12595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:32:15.868997 12663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:32:15.874166 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 12595
I20260926 11:32:15.874306 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:32:15.874605 392 external_mini_cluster.cc:1428] Setting key e42b130336be4381c4480ca5e21542ed
I20260926 11:32:15.874807 12663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a: Bootstrap starting.
I20260926 11:32:15.877731 12663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a: Neither blocks nor log segments found. Creating new log.
I20260926 11:32:15.878654 12663 log.cc:824] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a: Log is configured to *not* fsync() on all Append() calls
I20260926 11:32:15.881124 12663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a: No bootstrap required, opened a new log
Sep 26 11:32:15 dist-test-slave-mrck krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:32:15.886029 12663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "272dc9d37b684ad7aa3fd9fca08ea68a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33801 } }
I20260926 11:32:15.886319 12663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:32:15.886410 12663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 272dc9d37b684ad7aa3fd9fca08ea68a, State: Initialized, Role: FOLLOWER
I20260926 11:32:15.887039 12663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "272dc9d37b684ad7aa3fd9fca08ea68a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33801 } }
I20260926 11:32:15.887238 12663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:32:15.887300 12663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:32:15.887425 12663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:32:15.888661 12663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "272dc9d37b684ad7aa3fd9fca08ea68a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33801 } }
I20260926 11:32:15.889097 12663 leader_election.cc:302] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 272dc9d37b684ad7aa3fd9fca08ea68a; no voters:
I20260926 11:32:15.889459 12663 leader_election.cc:288] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:32:15.889622 12668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:32:15.889883 12668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [term 1 LEADER]: Becoming Leader. State: Replica: 272dc9d37b684ad7aa3fd9fca08ea68a, State: Running, Role: LEADER
I20260926 11:32:15.890003 12666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:15.876188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46928 (local address 127.0.98.62:33801)
0926 11:32:15.876691 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:15.876710 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:15.876753 (+ 43us) server_negotiation.cc:407] Connection header received
0926 11:32:15.877606 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:15.877632 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:15.878108 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:15.878481 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:15.879345 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:15.880448 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:15.881393 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:15.881878 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:15.882051 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:15.884718 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:15.884723 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:15.884738 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:15.884764 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:15.886605 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:15.887062 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:15.887070 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:15.887080 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:15.887343 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:15.887672 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:15.887676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:15.887677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:15.888160 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:15.888389 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:15.888637 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 11:32:15.889006 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":159,"threads_started":1}
I20260926 11:32:15.890265 12668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "272dc9d37b684ad7aa3fd9fca08ea68a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33801 } }
I20260926 11:32:15.890831 12663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:32:15.891561 12668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 272dc9d37b684ad7aa3fd9fca08ea68a. Latest consensus state: current_term: 1 leader_uuid: "272dc9d37b684ad7aa3fd9fca08ea68a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "272dc9d37b684ad7aa3fd9fca08ea68a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33801 } } }
I20260926 11:32:15.891659 12668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [sys.catalog]: This master's current role is: LEADER
I20260926 11:32:15.892455 12670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "272dc9d37b684ad7aa3fd9fca08ea68a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "272dc9d37b684ad7aa3fd9fca08ea68a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33801 } } }
I20260926 11:32:15.892609 12670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [sys.catalog]: This master's current role is: LEADER
I20260926 11:32:15.893276 12674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:32:15.893471 12663 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 11:32:15.899797 12674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:32:15.912923 12674 catalog_manager.cc:1347] Generated new cluster ID: 45b17c1d4aaf40ceadddaeb2ebe1e92d
I20260926 11:32:15.913004 12674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:32:15.941423 12674 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:32:15.943002 12674 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:32:15.956342 12674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a: Generated new TSK 0
I20260926 11:32:15.957413 12674 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:32:16.663868 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58293/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33801
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:32:16.819007 12708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:32:16.819286 12708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:32:16.819420 12708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:32:16.823987 12708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:32:16.824069 12708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:32:16.824194 12708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:32:16.831173 12708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:58293/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:32:16.832466 12708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:32:16.833582 12708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:32:16.844597 12714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:16.844597 12716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:16.845124 12713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:16.845320 12708 server_base.cc:1034] running on GCE node
I20260926 11:32:16.845814 12708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:32:16.846493 12708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:32:16.847704 12708 hybrid_clock.cc:648] HybridClock initialized: now 1790422336847670 us; error 45 us; skew 500 ppm
Sep 26 11:32:16 dist-test-slave-mrck krb5kdc[12151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422336, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:32:16.850533 12708 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:32:16.851619 12708 webserver.cc:460] Webserver started at http://127.0.98.1:37231/ using document root <none> and password file <none>
I20260926 11:32:16.852363 12708 fs_manager.cc:359] Metadata directory not provided
I20260926 11:32:16.852492 12708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:32:16.852761 12708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 11:32:16 dist-test-slave-mrck krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422336, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:32:16.869601 12708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-0/data/instance:
uuid: "f5d0dd97f1b94d5e9b7fd8ccca5b8ee0"
format_stamp: "Formatted at 2026-09-26 11:32:16 on dist-test-slave-mrck"
server_key: "cdcb5a3d69c18d5be8842e8784c04b8a"
server_key_iv: "527282b120e1915fc9c944baa1f28458"
server_key_version: "kuduclusterkey@0"
I20260926 11:32:16.870270 12708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f5d0dd97f1b94d5e9b7fd8ccca5b8ee0"
format_stamp: "Formatted at 2026-09-26 11:32:16 on dist-test-slave-mrck"
server_key: "cdcb5a3d69c18d5be8842e8784c04b8a"
server_key_iv: "527282b120e1915fc9c944baa1f28458"
server_key_version: "kuduclusterkey@0"
I20260926 11:32:16.883006 12708 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.006s sys 0.000s
I20260926 11:32:16.901278 12725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:16.903026 12708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:32:16.903153 12708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "f5d0dd97f1b94d5e9b7fd8ccca5b8ee0"
format_stamp: "Formatted at 2026-09-26 11:32:16 on dist-test-slave-mrck"
server_key: "cdcb5a3d69c18d5be8842e8784c04b8a"
server_key_iv: "527282b120e1915fc9c944baa1f28458"
server_key_version: "kuduclusterkey@0"
I20260926 11:32:16.903286 12708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:32:16.926594 12708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:32:16.930507 12708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:32:16.930744 12708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:32:16.931459 12708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:32:16.932636 12708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:32:16.932729 12708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:16.932817 12708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:32:16.932871 12708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:16.943154 12708 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40917
I20260926 11:32:16.944408 12708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:32:16.951293 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 12708
I20260926 11:32:16.951436 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:32:16.951784 392 external_mini_cluster.cc:1428] Setting key e7e1701743eba771c2ae04adaeea61a0
I20260926 11:32:16.960608 12838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40917 every 8 connection(s)
Sep 26 11:32:16 dist-test-slave-mrck krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422336, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:32:16.975050 12843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:16.959938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38939 (local address 127.0.98.62:33801)
0926 11:32:16.962551 (+ 2613us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:16.962555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:16.962569 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:32:16.962611 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:16.962614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:16.962660 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:16.962746 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:16.964271 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:16.965175 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:16.966879 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:16.967269 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:16.967349 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:16.970547 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:16.970552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:16.970555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:16.970584 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:16.972509 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:16.973170 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:16.973174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:16.973176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:16.973267 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:16.973689 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:16.973693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:16.973695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:16.973922 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:16.974007 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:16.974723 (+ 716us) server_negotiation.cc:299] Negotiation successful
0926 11:32:16.974862 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2532,"thread_start_us":67,"threads_started":1}
I20260926 11:32:16.977110 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:16.958993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33801 (local address 127.0.98.1:38939)
0926 11:32:16.961031 (+ 2038us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:16.961082 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:16.962116 (+ 1034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:16.963044 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:16.963053 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:16.963409 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:32:16.964099 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:16.964120 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:16.965326 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:16.965329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:16.966737 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:16.966748 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:16.967575 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:16.967579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:16.967811 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:16.968521 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:32:16.968542 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:32:16.970310 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:32:16.972661 (+ 2351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:16.972667 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:16.972685 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:16.973048 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:16.973375 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:16.973378 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:16.973380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:16.973577 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:16.974075 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:32:16.974083 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:32:16.974424 (+ 341us) client_negotiation.cc:769] Sending connection context
0926 11:32:16.975550 (+ 1126us) client_negotiation.cc:240] Negotiation successful
0926 11:32:16.976002 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1807,"thread_start_us":138,"threads_started":1}
I20260926 11:32:16.978900 12839 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33801
I20260926 11:32:16.979147 12839 heartbeater.cc:464] Registering TS with master...
I20260926 11:32:16.979720 12839 heartbeater.cc:511] Master 127.0.98.62:33801 requested a full tablet report, sending...
I20260926 11:32:16.981738 12627 ts_manager.cc:194] Registered new tserver with Master: f5d0dd97f1b94d5e9b7fd8ccca5b8ee0 (127.0.98.1:40917)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 11:32:16.983309 12627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38939
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:32:17.015051 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58293/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33801
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 11:32:17 dist-test-slave-mrck krb5kdc[12151](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790422337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 11:32:17.181447 12847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:32:17.181864 12847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:32:17.181936 12847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:32:17.188215 12847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:32:17.188305 12847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:32:17.188391 12847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:32:17.195405 12847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:58293/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:32:17.196607 12847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:32:17.197649 12847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:32:17.208782 12853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:17.208878 12856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:17.209064 12854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:17.209075 12847 server_base.cc:1034] running on GCE node
I20260926 11:32:17.209746 12847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:32:17.210510 12847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:32:17.211702 12847 hybrid_clock.cc:648] HybridClock initialized: now 1790422337211680 us; error 38 us; skew 500 ppm
Sep 26 11:32:17 dist-test-slave-mrck krb5kdc[12151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:32:17.215209 12847 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:32:17.216346 12847 webserver.cc:460] Webserver started at http://127.0.98.2:33457/ using document root <none> and password file <none>
I20260926 11:32:17.217048 12847 fs_manager.cc:359] Metadata directory not provided
I20260926 11:32:17.217140 12847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:32:17.217447 12847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 11:32:17 dist-test-slave-mrck krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:32:17.232679 12847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-1/data/instance:
uuid: "8f2ec2bd33ad480781c059a23173347d"
format_stamp: "Formatted at 2026-09-26 11:32:17 on dist-test-slave-mrck"
server_key: "89754b3316a85e1a168f0812466efaf8"
server_key_iv: "acaed8d42ec665cc92b0ed448a418124"
server_key_version: "kuduclusterkey@0"
I20260926 11:32:17.233585 12847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8f2ec2bd33ad480781c059a23173347d"
format_stamp: "Formatted at 2026-09-26 11:32:17 on dist-test-slave-mrck"
server_key: "89754b3316a85e1a168f0812466efaf8"
server_key_iv: "acaed8d42ec665cc92b0ed448a418124"
server_key_version: "kuduclusterkey@0"
I20260926 11:32:17.238696 12847 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 11:32:17.274121 12863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:17.275978 12847 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.004s
I20260926 11:32:17.276120 12847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "8f2ec2bd33ad480781c059a23173347d"
format_stamp: "Formatted at 2026-09-26 11:32:17 on dist-test-slave-mrck"
server_key: "89754b3316a85e1a168f0812466efaf8"
server_key_iv: "acaed8d42ec665cc92b0ed448a418124"
server_key_version: "kuduclusterkey@0"
I20260926 11:32:17.276260 12847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:32:17.296058 12847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:32:17.299772 12847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:32:17.299952 12847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:32:17.300624 12847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:32:17.301672 12847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:32:17.301776 12847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:17.301872 12847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:32:17.301932 12847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:17.311681 12847 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42587
I20260926 11:32:17.312907 12847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:32:17.318531 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 12847
I20260926 11:32:17.318666 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:32:17.318972 392 external_mini_cluster.cc:1428] Setting key a35f61193c8274303ca522386c44d0d2
I20260926 11:32:17.323655 12976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42587 every 8 connection(s)
Sep 26 11:32:17 dist-test-slave-mrck krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 11:32:17.331350 12843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:17.315160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53183 (local address 127.0.98.62:33801)
0926 11:32:17.315305 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:17.315310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:17.316401 (+ 1091us) server_negotiation.cc:407] Connection header received
0926 11:32:17.317715 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:17.317721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:17.317781 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:17.317842 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:17.319866 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:17.320635 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:17.322499 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:17.322898 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:17.322991 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:17.326077 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:17.326081 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:17.326084 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:17.326113 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:17.328553 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:17.329277 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:17.329282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:17.329284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:17.329372 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:17.329811 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:17.329817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:17.329820 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:17.330092 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:17.330190 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:17.330950 (+ 760us) server_negotiation.cc:299] Negotiation successful
0926 11:32:17.331107 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 11:32:17.332916 12979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:17.315658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33801 (local address 127.0.98.2:53183)
0926 11:32:17.316232 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:17.316292 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:17.317428 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:17.318301 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:17.318311 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:17.318699 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:32:17.319653 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:17.319677 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:17.320795 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:17.320801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:17.322342 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:17.322356 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:17.323214 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:17.323218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:17.323422 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:17.324111 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:32:17.324131 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:32:17.325843 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:32:17.328720 (+ 2877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:17.328726 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:17.328743 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:17.329146 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:17.329491 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:17.329498 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:17.329501 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:17.329699 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:17.330268 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:32:17.330278 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:32:17.330650 (+ 372us) client_negotiation.cc:769] Sending connection context
0926 11:32:17.331551 (+ 901us) client_negotiation.cc:240] Negotiation successful
0926 11:32:17.331960 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":134,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 11:32:17.334355 12977 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33801
I20260926 11:32:17.334606 12977 heartbeater.cc:464] Registering TS with master...
I20260926 11:32:17.335137 12977 heartbeater.cc:511] Master 127.0.98.62:33801 requested a full tablet report, sending...
I20260926 11:32:17.337322 12627 ts_manager.cc:194] Registered new tserver with Master: 8f2ec2bd33ad480781c059a23173347d (127.0.98.2:42587)
I20260926 11:32:17.338192 12627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:32:17.353085 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58293/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33801
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:32:17.520323 12982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:32:17.520694 12982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:32:17.520823 12982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:32:17.526983 12982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:32:17.527180 12982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:32:17.527333 12982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:32:17.535506 12982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:58293/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:32:17.537237 12982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:32:17.538671 12982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:32:17.551770 12988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
Sep 26 11:32:17 dist-test-slave-mrck krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:32:17.553267 12982 server_base.cc:1034] running on GCE node
W20260926 11:32:17.551919 12987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:17.552165 12990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:17.554185 12982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:32:17.554956 12982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:32:17.556180 12982 hybrid_clock.cc:648] HybridClock initialized: now 1790422337556133 us; error 48 us; skew 500 ppm
Sep 26 11:32:17 dist-test-slave-mrck krb5kdc[12151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:32:17.560402 12982 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:32:17.561551 12982 webserver.cc:460] Webserver started at http://127.0.98.3:35127/ using document root <none> and password file <none>
I20260926 11:32:17.562300 12982 fs_manager.cc:359] Metadata directory not provided
I20260926 11:32:17.562412 12982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:32:17.562702 12982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 11:32:17 dist-test-slave-mrck krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:32:17.585834 12982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-2/data/instance:
uuid: "e97d17d5aeb14ef88b88b9db9d7d08b3"
format_stamp: "Formatted at 2026-09-26 11:32:17 on dist-test-slave-mrck"
server_key: "cec5a030650b46d2ca6783677c3625a9"
server_key_iv: "cbea32593b279864cef81c20a5cc84d4"
server_key_version: "kuduclusterkey@0"
I20260926 11:32:17.586839 12982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e97d17d5aeb14ef88b88b9db9d7d08b3"
format_stamp: "Formatted at 2026-09-26 11:32:17 on dist-test-slave-mrck"
server_key: "cec5a030650b46d2ca6783677c3625a9"
server_key_iv: "cbea32593b279864cef81c20a5cc84d4"
server_key_version: "kuduclusterkey@0"
I20260926 11:32:17.603284 12982 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.004s sys 0.000s
I20260926 11:32:17.622658 12997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:17.624426 12982 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 11:32:17.624577 12982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "e97d17d5aeb14ef88b88b9db9d7d08b3"
format_stamp: "Formatted at 2026-09-26 11:32:17 on dist-test-slave-mrck"
server_key: "cec5a030650b46d2ca6783677c3625a9"
server_key_iv: "cbea32593b279864cef81c20a5cc84d4"
server_key_version: "kuduclusterkey@0"
I20260926 11:32:17.624715 12982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:32:17.644208 12982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:32:17.648262 12982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:32:17.648530 12982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:32:17.649293 12982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:32:17.650395 12982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:32:17.650501 12982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:17.650611 12982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:32:17.650679 12982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:17.661619 12982 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45173
I20260926 11:32:17.662942 12982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:32:17.672272 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 12982
I20260926 11:32:17.672295 13110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45173 every 8 connection(s)
I20260926 11:32:17.672421 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:32:17.672739 392 external_mini_cluster.cc:1428] Setting key e4ef8a1a4f216cf8e04da94d561c0f83
Sep 26 11:32:17 dist-test-slave-mrck krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:32:17.689296 12843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:17.675326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51375 (local address 127.0.98.62:33801)
0926 11:32:17.675882 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:17.675887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:17.677066 (+ 1179us) server_negotiation.cc:407] Connection header received
0926 11:32:17.678179 (+ 1113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:17.678181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:17.678226 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:17.678296 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:17.679664 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:17.680257 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:17.681841 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:17.682119 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:17.682210 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:17.685020 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:17.685024 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:17.685026 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:17.685051 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:17.686886 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:17.687490 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:17.687494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:17.687496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:17.687600 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:17.688089 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:17.688093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:17.688095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:17.688328 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:17.688431 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:17.689010 (+ 579us) server_negotiation.cc:299] Negotiation successful
0926 11:32:17.689140 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465}
I20260926 11:32:17.690145 13113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:17.675876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33801 (local address 127.0.98.3:51375)
0926 11:32:17.676877 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:17.676937 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:17.677954 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:17.678482 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:17.678491 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:17.678851 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:32:17.679505 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:17.679523 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:17.680384 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:17.680387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:17.681726 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:17.681737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:17.682282 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:17.682286 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:17.682481 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:17.683248 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:32:17.683268 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:32:17.684831 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:32:17.687004 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:17.687011 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:17.687029 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:17.687392 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:17.687725 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:17.687785 (+ 60us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:17.687790 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:17.687975 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:17.688447 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:32:17.688453 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:32:17.688733 (+ 280us) client_negotiation.cc:769] Sending connection context
0926 11:32:17.688912 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 11:32:17.689185 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"thread_start_us":155,"threads_started":1}
I20260926 11:32:17.691401 13111 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33801
I20260926 11:32:17.691674 13111 heartbeater.cc:464] Registering TS with master...
I20260926 11:32:17.692327 13111 heartbeater.cc:511] Master 127.0.98.62:33801 requested a full tablet report, sending...
I20260926 11:32:17.693473 12627 ts_manager.cc:194] Registered new tserver with Master: e97d17d5aeb14ef88b88b9db9d7d08b3 (127.0.98.3:45173)
I20260926 11:32:17.694085 12627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51375
I20260926 11:32:17.699095 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:32:17.699895 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 12708
I20260926 11:32:17.705651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 12847
I20260926 11:32:17.711410 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 12982
I20260926 11:32:17.718355 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 12595
I20260926 11:32:17.724961 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33801
--webserver_interface=127.0.98.62
--webserver_port=37153
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33801
--ranger_config_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58293/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:58293/kms with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:32:17.835520 13114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:32:17.835873 13114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:32:17.835952 13114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:32:17.840013 13114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:32:17.840070 13114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:32:17.840090 13114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:32:17.840106 13114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:32:17.840121 13114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:32:17.846279 13114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:58293/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33801
--ranger_config_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37153
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:32:17.847406 13114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:32:17.848527 13114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:32:17.856985 13119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:17.856985 13120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:17.857089 13114 server_base.cc:1034] running on GCE node
W20260926 11:32:17.857023 13122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:17.857822 13114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:32:17.858963 13114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:32:17.860152 13114 hybrid_clock.cc:648] HybridClock initialized: now 1790422337860121 us; error 44 us; skew 500 ppm
Sep 26 11:32:17 dist-test-slave-mrck krb5kdc[12151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:32:17.862804 13114 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:32:17.863687 13114 webserver.cc:460] Webserver started at http://127.0.98.62:37153/ using document root <none> and password file <none>
I20260926 11:32:17.864264 13114 fs_manager.cc:359] Metadata directory not provided
I20260926 11:32:17.864362 13114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 11:32:17 dist-test-slave-mrck krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:32:17.933686 13114 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260926 11:32:17.936039 13130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:17.937309 13114 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 11:32:17.937459 13114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "272dc9d37b684ad7aa3fd9fca08ea68a"
format_stamp: "Formatted at 2026-09-26 11:32:15 on dist-test-slave-mrck"
server_key: "ce0139291c9469abee62268fc83f68c7"
server_key_iv: "f3b7ab46c6deca421e526062af330425"
server_key_version: "kuduclusterkey@0"
I20260926 11:32:17.938024 13114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:32:17.953724 13114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:32:17.956826 13114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:32:17.957041 13114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:32:17.964416 13114 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33801
I20260926 11:32:17.964419 13182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33801 every 8 connection(s)
I20260926 11:32:17.965463 13114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:32:17.969745 13183 sys_catalog.cc:263] Verifying existing consensus state
I20260926 11:32:17.971316 13183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a: Bootstrap starting.
I20260926 11:32:17.972435 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 13114
I20260926 11:32:17.972862 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58293/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40917
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33801
--webserver_port=37231
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:58293/kms with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:32:17.988346 13183 log.cc:824] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a: Log is configured to *not* fsync() on all Append() calls
I20260926 11:32:17.995254 13183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 11:32:17.995653 13183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a: Bootstrap complete.
I20260926 11:32:17.999897 13183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "272dc9d37b684ad7aa3fd9fca08ea68a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33801 } }
I20260926 11:32:18.000340 13183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 272dc9d37b684ad7aa3fd9fca08ea68a, State: Initialized, Role: FOLLOWER
I20260926 11:32:18.000869 13183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "272dc9d37b684ad7aa3fd9fca08ea68a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33801 } }
I20260926 11:32:18.001027 13183 raft_consensus.cc:388] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:32:18.001111 13183 raft_consensus.cc:482] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:32:18.001222 13183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [term 1 FOLLOWER]: Advancing to term 2
I20260926 11:32:18.003124 13183 raft_consensus.cc:504] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "272dc9d37b684ad7aa3fd9fca08ea68a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33801 } }
I20260926 11:32:18.003463 13183 leader_election.cc:302] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 272dc9d37b684ad7aa3fd9fca08ea68a; no voters:
I20260926 11:32:18.003820 13183 leader_election.cc:288] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 11:32:18.003906 13187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [term 2 FOLLOWER]: Leader election won for term 2
I20260926 11:32:18.004179 13187 raft_consensus.cc:686] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [term 2 LEADER]: Becoming Leader. State: Replica: 272dc9d37b684ad7aa3fd9fca08ea68a, State: Running, Role: LEADER
I20260926 11:32:18.004570 13187 consensus_queue.cc:237] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "272dc9d37b684ad7aa3fd9fca08ea68a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33801 } }
I20260926 11:32:18.005517 13183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:32:18.005908 13189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "272dc9d37b684ad7aa3fd9fca08ea68a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "272dc9d37b684ad7aa3fd9fca08ea68a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33801 } } }
I20260926 11:32:18.006034 13189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [sys.catalog]: This master's current role is: LEADER
I20260926 11:32:18.006351 13188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 272dc9d37b684ad7aa3fd9fca08ea68a. Latest consensus state: current_term: 2 leader_uuid: "272dc9d37b684ad7aa3fd9fca08ea68a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "272dc9d37b684ad7aa3fd9fca08ea68a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33801 } } }
I20260926 11:32:18.006459 13188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a [sys.catalog]: This master's current role is: LEADER
I20260926 11:32:18.006971 13192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:32:18.010223 13192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:32:18.010774 13183 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 11:32:18.010882 13192 catalog_manager.cc:1259] Loaded cluster ID: 45b17c1d4aaf40ceadddaeb2ebe1e92d
I20260926 11:32:18.010931 13192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:32:18.016021 13192 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:32:18.017336 13192 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a: Loaded TSK: 0
I20260926 11:32:18.018743 13192 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 11:32:18.116874 13185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:32:18.117202 13185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:32:18.117297 13185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:32:18.121951 13185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:32:18.122071 13185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:32:18.122172 13185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:32:18.129519 13185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:58293/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40917
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37231
--tserver_master_addrs=127.0.98.62:33801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:32:18.130817 13185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:32:18.132201 13185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:32:18.142042 13221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:18.142117 13224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:18.142405 13185 server_base.cc:1034] running on GCE node
W20260926 11:32:18.142637 13222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:18.143078 13185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:32:18.143795 13185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:32:18.144994 13185 hybrid_clock.cc:648] HybridClock initialized: now 1790422338144960 us; error 46 us; skew 500 ppm
Sep 26 11:32:18 dist-test-slave-mrck krb5kdc[12151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:32:18.148752 13185 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:32:18.149863 13185 webserver.cc:460] Webserver started at http://127.0.98.1:37231/ using document root <none> and password file <none>
I20260926 11:32:18.150588 13185 fs_manager.cc:359] Metadata directory not provided
I20260926 11:32:18.150698 13185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 11:32:18 dist-test-slave-mrck krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:32:18.182477 13185 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.004s sys 0.000s
I20260926 11:32:18.186113 13232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:18.187904 13185 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 11:32:18.188043 13185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "f5d0dd97f1b94d5e9b7fd8ccca5b8ee0"
format_stamp: "Formatted at 2026-09-26 11:32:16 on dist-test-slave-mrck"
server_key: "cdcb5a3d69c18d5be8842e8784c04b8a"
server_key_iv: "527282b120e1915fc9c944baa1f28458"
server_key_version: "kuduclusterkey@0"
I20260926 11:32:18.188798 13185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:32:18.220547 13185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:32:18.224455 13185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:32:18.224684 13185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:32:18.225380 13185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:32:18.226416 13185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:32:18.226483 13185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:18.226536 13185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:32:18.226560 13185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:18.237479 13185 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40917
I20260926 11:32:18.238842 13185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:32:18.239347 13345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40917 every 8 connection(s)
I20260926 11:32:18.243140 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 13185
I20260926 11:32:18.243654 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58293/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42587
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33801
--webserver_port=33457
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:58293/kms with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 11:32:18 dist-test-slave-mrck krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:32:18.262265 13348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:18.242029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33801 (local address 127.0.98.1:42685)
0926 11:32:18.242620 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:18.242678 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:18.244868 (+ 2190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:18.246821 (+ 1953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:18.246829 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:18.247218 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:32:18.247880 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:18.247901 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:18.249766 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:18.249773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:18.251323 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:18.251335 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:18.252800 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:18.252807 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:18.253037 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:18.253899 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:32:18.253921 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:32:18.255702 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:32:18.258495 (+ 2793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:18.258504 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:18.258523 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:18.258893 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:18.259361 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:18.259364 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:18.259366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:18.259548 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:18.260341 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:32:18.260348 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:32:18.260654 (+ 306us) client_negotiation.cc:769] Sending connection context
0926 11:32:18.260853 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 11:32:18.261209 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":149,"threads_started":1}
I20260926 11:32:18.264132 13349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:18.243426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42685 (local address 127.0.98.62:33801)
0926 11:32:18.243861 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:18.243882 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:18.243930 (+ 48us) server_negotiation.cc:407] Connection header received
0926 11:32:18.245483 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:18.245499 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:18.245920 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:18.246322 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:18.248071 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:18.249602 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:18.251606 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:18.252372 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:18.252565 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:18.255987 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:18.255994 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:18.256012 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:18.256046 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:18.258327 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:18.259056 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:18.259063 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:18.259071 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:18.259247 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:18.259660 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:18.259664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:18.259666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:18.260159 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:18.260288 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:18.262860 (+ 2572us) server_negotiation.cc:299] Negotiation successful
0926 11:32:18.263161 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":127,"threads_started":1}
I20260926 11:32:18.267689 13346 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33801
I20260926 11:32:18.267962 13346 heartbeater.cc:464] Registering TS with master...
I20260926 11:32:18.268555 13346 heartbeater.cc:511] Master 127.0.98.62:33801 requested a full tablet report, sending...
W20260926 11:32:18.270867 13346 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33801 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 11:32:18.271003 13346 heartbeater.cc:464] Registering TS with master...
I20260926 11:32:18.271212 13346 heartbeater.cc:511] Master 127.0.98.62:33801 requested a full tablet report, sending...
I20260926 11:32:18.272110 13346 heartbeater.cc:464] Registering TS with master...
I20260926 11:32:18.272284 13346 heartbeater.cc:511] Master 127.0.98.62:33801 requested a full tablet report, sending...
W20260926 11:32:18.273011 13346 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 11:32:18.421375 13350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:32:18.421859 13350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:32:18.421975 13350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:32:18.427944 13350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:32:18.428026 13350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:32:18.428112 13350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:32:18.436096 13350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:58293/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42587
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33457
--tserver_master_addrs=127.0.98.62:33801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:32:18.437474 13350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:32:18.438804 13350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:32:18.451378 13356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:18.451383 13355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:18.451864 13358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:18.452744 13350 server_base.cc:1034] running on GCE node
I20260926 11:32:18.453313 13350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:32:18.454224 13350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:32:18.455436 13350 hybrid_clock.cc:648] HybridClock initialized: now 1790422338455408 us; error 46 us; skew 500 ppm
Sep 26 11:32:18 dist-test-slave-mrck krb5kdc[12151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:32:18.459074 13350 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:32:18.460479 13350 webserver.cc:460] Webserver started at http://127.0.98.2:33457/ using document root <none> and password file <none>
I20260926 11:32:18.461179 13350 fs_manager.cc:359] Metadata directory not provided
I20260926 11:32:18.461265 13350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 11:32:18 dist-test-slave-mrck krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:32:18.483775 13350 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260926 11:32:18.493116 13366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:18.494902 13350 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.000s
I20260926 11:32:18.495031 13350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "8f2ec2bd33ad480781c059a23173347d"
format_stamp: "Formatted at 2026-09-26 11:32:17 on dist-test-slave-mrck"
server_key: "89754b3316a85e1a168f0812466efaf8"
server_key_iv: "acaed8d42ec665cc92b0ed448a418124"
server_key_version: "kuduclusterkey@0"
I20260926 11:32:18.495806 13350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:32:18.519797 13350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:32:18.523416 13350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:32:18.523751 13350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:32:18.524549 13350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:32:18.525611 13350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:32:18.525712 13350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:18.525807 13350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:32:18.525867 13350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:18.537623 13350 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42587
I20260926 11:32:18.539013 13350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:32:18.540021 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 13350
I20260926 11:32:18.540575 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:58293/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45173
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33801
--webserver_port=35127
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:58293/kms with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:32:18.543339 13479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42587 every 8 connection(s)
Sep 26 11:32:18 dist-test-slave-mrck krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:32:18.557668 13349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:18.543367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48111 (local address 127.0.98.62:33801)
0926 11:32:18.543522 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:18.543529 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:18.544046 (+ 517us) server_negotiation.cc:407] Connection header received
0926 11:32:18.545232 (+ 1186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:18.545236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:18.545288 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:18.545340 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:18.546965 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:18.547802 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:18.549435 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:18.549784 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:18.549857 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:18.553538 (+ 3681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:18.553543 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:18.553545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:18.553575 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:18.555264 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:18.555907 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:18.555911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:18.555913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:18.555996 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:18.556366 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:18.556370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:18.556371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:18.556577 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:18.556672 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:18.557350 (+ 678us) server_negotiation.cc:299] Negotiation successful
0926 11:32:18.557484 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 11:32:18.559332 13483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:18.542265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33801 (local address 127.0.98.2:48111)
0926 11:32:18.543867 (+ 1602us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:18.543921 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:18.544922 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:18.545757 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:18.545766 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:18.546100 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:32:18.546792 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:18.546812 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:18.547963 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:18.547967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:18.549295 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:18.549306 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:18.550059 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:18.550063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:18.550257 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:18.550960 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:32:18.550980 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:32:18.553331 (+ 2351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:32:18.555407 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:18.555413 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:18.555431 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:18.555794 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:18.556099 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:18.556102 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:18.556104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:18.556266 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:18.556740 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:32:18.556747 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:32:18.557075 (+ 328us) client_negotiation.cc:769] Sending connection context
0926 11:32:18.557917 (+ 842us) client_negotiation.cc:240] Negotiation successful
0926 11:32:18.558265 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1373,"thread_start_us":122,"threads_started":1}
I20260926 11:32:18.560701 13480 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33801
I20260926 11:32:18.560953 13480 heartbeater.cc:464] Registering TS with master...
I20260926 11:32:18.561535 13480 heartbeater.cc:511] Master 127.0.98.62:33801 requested a full tablet report, sending...
W20260926 11:32:18.565363 13480 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33801 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 11:32:18.565501 13480 heartbeater.cc:464] Registering TS with master...
I20260926 11:32:18.565706 13480 heartbeater.cc:511] Master 127.0.98.62:33801 requested a full tablet report, sending...
I20260926 11:32:18.566521 13480 heartbeater.cc:464] Registering TS with master...
I20260926 11:32:18.566712 13480 heartbeater.cc:511] Master 127.0.98.62:33801 requested a full tablet report, sending...
W20260926 11:32:18.567452 13480 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 11:32:18.721951 13482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:32:18.722216 13482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:32:18.722261 13482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:32:18.726739 13482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:32:18.726847 13482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:32:18.726958 13482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:32:18.733917 13482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:58293/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35127
--tserver_master_addrs=127.0.98.62:33801
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:32:18.735357 13482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:32:18.736826 13482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:32:18.749346 13490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:18.749336 13489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:18.749667 13482 server_base.cc:1034] running on GCE node
W20260926 11:32:18.749912 13492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:18.750339 13482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:32:18.751205 13482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:32:18.752384 13482 hybrid_clock.cc:648] HybridClock initialized: now 1790422338752354 us; error 41 us; skew 500 ppm
Sep 26 11:32:18 dist-test-slave-mrck krb5kdc[12151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:32:18.755450 13482 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:32:18.756456 13482 webserver.cc:460] Webserver started at http://127.0.98.3:35127/ using document root <none> and password file <none>
I20260926 11:32:18.757030 13482 fs_manager.cc:359] Metadata directory not provided
I20260926 11:32:18.757103 13482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 11:32:18 dist-test-slave-mrck krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:32:18.774821 13482 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 11:32:18.778532 13500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:18.780356 13482 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 11:32:18.780603 13482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "e97d17d5aeb14ef88b88b9db9d7d08b3"
format_stamp: "Formatted at 2026-09-26 11:32:17 on dist-test-slave-mrck"
server_key: "cec5a030650b46d2ca6783677c3625a9"
server_key_iv: "cbea32593b279864cef81c20a5cc84d4"
server_key_version: "kuduclusterkey@0"
I20260926 11:32:18.781365 13482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:32:18.803071 13482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:32:18.806842 13482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:32:18.807049 13482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:32:18.807816 13482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:32:18.808857 13482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:32:18.808925 13482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:18.808983 13482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:32:18.809021 13482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:18.820575 13482 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45173
I20260926 11:32:18.820649 13613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45173 every 8 connection(s)
I20260926 11:32:18.821905 13482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:32:18.832167 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 13482
Sep 26 11:32:18 dist-test-slave-mrck krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:32:18.840201 13349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:18.824089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41699 (local address 127.0.98.62:33801)
0926 11:32:18.824227 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:18.824232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:18.825394 (+ 1162us) server_negotiation.cc:407] Connection header received
0926 11:32:18.826672 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:18.826675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:18.826735 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:18.826840 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:18.828343 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:18.829179 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:18.830908 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:18.831262 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:18.831386 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:18.835406 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:18.835410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:18.835413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:18.835440 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:18.837344 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:18.838017 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:18.838021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:18.838023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:18.838121 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:18.838558 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:18.838562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:18.838563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:18.838750 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:18.838864 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:18.839890 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0926 11:32:18.840022 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 11:32:18.840922 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:18.824618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33801 (local address 127.0.98.3:41699)
0926 11:32:18.825222 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:18.825278 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:18.826404 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:18.827053 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:18.827063 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:18.827443 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:32:18.828163 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:18.828185 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:18.829335 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:18.829339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:18.830761 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:18.830772 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:18.831444 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:18.831448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:18.831666 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:18.833476 (+ 1810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:32:18.833505 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:32:18.835252 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:32:18.837464 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:18.837470 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:18.837488 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:18.837896 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:18.838239 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:18.838244 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:18.838246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:18.838437 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:18.838861 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:32:18.838868 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:32:18.839254 (+ 386us) client_negotiation.cc:769] Sending connection context
0926 11:32:18.839501 (+ 247us) client_negotiation.cc:240] Negotiation successful
0926 11:32:18.839894 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":162,"threads_started":1}
I20260926 11:32:18.842499 13614 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33801
I20260926 11:32:18.842757 13614 heartbeater.cc:464] Registering TS with master...
I20260926 11:32:18.843354 13614 heartbeater.cc:511] Master 127.0.98.62:33801 requested a full tablet report, sending...
W20260926 11:32:18.845160 13614 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33801 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 11:32:18.845306 13614 heartbeater.cc:464] Registering TS with master...
I20260926 11:32:18.845497 13614 heartbeater.cc:511] Master 127.0.98.62:33801 requested a full tablet report, sending...
I20260926 11:32:18.846344 13614 heartbeater.cc:464] Registering TS with master...
I20260926 11:32:18.846361 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:18.832859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46944 (local address 127.0.98.62:33801)
0926 11:32:18.833233 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:18.833237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:18.833684 (+ 447us) server_negotiation.cc:407] Connection header received
0926 11:32:18.833724 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:18.833727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:18.833796 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:18.833886 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:18.834780 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:18.835833 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:18.837260 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:18.837599 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:18.837710 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:18.840402 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:18.840406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:18.840409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:18.840433 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:18.842608 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:18.844486 (+ 1878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:18.844490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:18.844492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:18.844593 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:18.844899 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:18.844902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:18.844904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:18.845136 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:18.845340 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:18.846047 (+ 707us) server_negotiation.cc:299] Negotiation successful
0926 11:32:18.846179 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":114,"threads_started":1}
I20260926 11:32:18.846571 13614 heartbeater.cc:511] Master 127.0.98.62:33801 requested a full tablet report, sending...
W20260926 11:32:18.848179 13614 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 11:32:19 dist-test-slave-mrck krb5kdc[12151](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790422339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:32:19.273263 13346 heartbeater.cc:464] Registering TS with master...
I20260926 11:32:19.273583 13346 heartbeater.cc:511] Master 127.0.98.62:33801 requested a full tablet report, sending...
I20260926 11:32:19.275036 13146 ts_manager.cc:194] Registered new tserver with Master: f5d0dd97f1b94d5e9b7fd8ccca5b8ee0 (127.0.98.1:40917)
I20260926 11:32:19.276767 13146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42685
I20260926 11:32:19.567777 13480 heartbeater.cc:464] Registering TS with master...
I20260926 11:32:19.568320 13480 heartbeater.cc:511] Master 127.0.98.62:33801 requested a full tablet report, sending...
I20260926 11:32:19.569249 13146 ts_manager.cc:194] Registered new tserver with Master: 8f2ec2bd33ad480781c059a23173347d (127.0.98.2:42587)
I20260926 11:32:19.570075 13146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48111
Sep 26 11:32:19 dist-test-slave-mrck krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:32:19.848417 13614 heartbeater.cc:464] Registering TS with master...
I20260926 11:32:19.848760 13614 heartbeater.cc:511] Master 127.0.98.62:33801 requested a full tablet report, sending...
I20260926 11:32:19.849604 13146 ts_manager.cc:194] Registered new tserver with Master: e97d17d5aeb14ef88b88b9db9d7d08b3 (127.0.98.3:45173)
I20260926 11:32:19.850364 13146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41699
I20260926 11:32:19.853792 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:32:19.867627 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:19.855381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60510 (local address 127.0.98.62:33801)
0926 11:32:19.855663 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:19.855668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:19.855970 (+ 302us) server_negotiation.cc:407] Connection header received
0926 11:32:19.858502 (+ 2532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:19.858506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:19.858559 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:19.858651 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:19.859555 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:19.860386 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:19.861502 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:19.861843 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:19.861924 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:19.863760 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:19.863764 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:19.863767 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:19.863793 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:19.865603 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:19.866146 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:19.866150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:19.866152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:19.866252 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:19.866572 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:19.866575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:19.866577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:19.866787 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:19.866895 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:19.867307 (+ 412us) server_negotiation.cc:299] Negotiation successful
0926 11:32:19.867444 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":102,"threads_started":1}
I20260926 11:32:19.872344 13146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:32:19.877048 13146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 26 11:32:19 dist-test-slave-mrck krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 11:32:19.910748 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:19.899927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40917 (local address 127.0.0.1:45478)
0926 11:32:19.904028 (+ 4101us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:19.904128 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:19.904272 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:19.905968 (+ 1696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:19.905971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:19.905987 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:32:19.906066 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:19.906072 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:19.907601 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:19.907606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:19.908805 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:19.908817 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:19.910156 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:19.910160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:19.910304 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:19.910343 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 11:32:19.910450 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 11:32:19.910570 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3956,"mutex_wait_us":5,"spinlock_wait_cycles":3584,"thread_start_us":78,"threads_started":1}
I20260926 11:32:19.911401 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:19.900283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45478 (local address 127.0.98.1:40917)
0926 11:32:19.904556 (+ 4273us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:19.904563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:19.904581 (+ 18us) server_negotiation.cc:407] Connection header received
0926 11:32:19.904642 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:19.904646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:19.905610 (+ 964us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:19.905811 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:32:19.906188 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:19.907436 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:19.908966 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:19.909820 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:19.909927 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:19.910092 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:19.911143 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0926 11:32:19.911257 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4170,"thread_start_us":81,"threads_started":1}
I20260926 11:32:19.914677 13269 tablet_service.cc:1467] Processing CreateTablet for tablet 46cee4dad78541e4b18661ab0982800d (DEFAULT_TABLE table=test-table [id=cc5c832cd4d14afaabffca817a1cc501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:32:19.916052 13269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46cee4dad78541e4b18661ab0982800d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:32:19.916421 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:19.901223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39138 (local address 127.0.98.3:45173)
0926 11:32:19.902311 (+ 1088us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:19.902319 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:19.907772 (+ 5453us) server_negotiation.cc:407] Connection header received
0926 11:32:19.908025 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:19.908030 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:19.908242 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:19.908349 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:32:19.910430 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:19.912057 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:19.913626 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:19.915438 (+ 1812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:19.915525 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:19.915660 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:19.916073 (+ 413us) server_negotiation.cc:299] Negotiation successful
0926 11:32:19.916279 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":980,"thread_start_us":111,"threads_started":1}
I20260926 11:32:19.916826 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:19.900975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45173 (local address 127.0.0.1:39138)
0926 11:32:19.907658 (+ 6683us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:19.907672 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:19.907873 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:19.908513 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:19.908515 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:19.910128 (+ 1613us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:32:19.910288 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:19.910295 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:19.912226 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:19.912231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:19.913480 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:19.913491 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:19.915809 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:19.915812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:19.915902 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:19.915936 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 11:32:19.916616 (+ 680us) client_negotiation.cc:240] Negotiation successful
0926 11:32:19.916699 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6620,"mutex_wait_us":55,"thread_start_us":86,"threads_started":1}
I20260926 11:32:19.919857 13548 tablet_service.cc:1467] Processing CreateTablet for tablet 46cee4dad78541e4b18661ab0982800d (DEFAULT_TABLE table=test-table [id=cc5c832cd4d14afaabffca817a1cc501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:32:19.921311 13548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46cee4dad78541e4b18661ab0982800d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:32:19.921329 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:19.898028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42587 (local address 127.0.0.1:37698)
0926 11:32:19.904028 (+ 6000us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:19.904129 (+ 101us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:19.904272 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:19.905111 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:19.905117 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:19.905138 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:32:19.905291 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:19.905299 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:19.906669 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:19.906672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:19.908024 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:19.908032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:19.920873 (+ 12841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:19.920876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:19.920981 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:19.921005 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 11:32:19.921108 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 11:32:19.921190 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5802,"spinlock_wait_cycles":7168,"thread_start_us":91,"threads_started":1}
I20260926 11:32:19.921382 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:19.898868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37698 (local address 127.0.98.2:42587)
0926 11:32:19.900673 (+ 1805us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:19.900681 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:19.904432 (+ 3751us) server_negotiation.cc:407] Connection header received
0926 11:32:19.904518 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:19.904524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:19.904756 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:19.904910 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:32:19.905437 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:19.906546 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:19.919839 (+ 13293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:19.920732 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:19.920852 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:19.920992 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:19.921146 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 11:32:19.921241 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1685,"thread_start_us":98,"threads_started":1}
I20260926 11:32:19.924153 13406 tablet_service.cc:1467] Processing CreateTablet for tablet 46cee4dad78541e4b18661ab0982800d (DEFAULT_TABLE table=test-table [id=cc5c832cd4d14afaabffca817a1cc501]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:32:19.925164 13406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46cee4dad78541e4b18661ab0982800d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:32:19.931301 13649 tablet_bootstrap.cc:492] T 46cee4dad78541e4b18661ab0982800d P f5d0dd97f1b94d5e9b7fd8ccca5b8ee0: Bootstrap starting.
I20260926 11:32:19.934996 13649 tablet_bootstrap.cc:654] T 46cee4dad78541e4b18661ab0982800d P f5d0dd97f1b94d5e9b7fd8ccca5b8ee0: Neither blocks nor log segments found. Creating new log.
I20260926 11:32:19.935923 13649 log.cc:824] T 46cee4dad78541e4b18661ab0982800d P f5d0dd97f1b94d5e9b7fd8ccca5b8ee0: Log is configured to *not* fsync() on all Append() calls
I20260926 11:32:19.938303 13649 tablet_bootstrap.cc:492] T 46cee4dad78541e4b18661ab0982800d P f5d0dd97f1b94d5e9b7fd8ccca5b8ee0: No bootstrap required, opened a new log
I20260926 11:32:19.938511 13649 ts_tablet_manager.cc:1397] T 46cee4dad78541e4b18661ab0982800d P f5d0dd97f1b94d5e9b7fd8ccca5b8ee0: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260926 11:32:19.943120 13649 raft_consensus.cc:348] T 46cee4dad78541e4b18661ab0982800d P f5d0dd97f1b94d5e9b7fd8ccca5b8ee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2ec2bd33ad480781c059a23173347d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42587 } } peers { permanent_uuid: "f5d0dd97f1b94d5e9b7fd8ccca5b8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40917 } } peers { permanent_uuid: "e97d17d5aeb14ef88b88b9db9d7d08b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45173 } }
I20260926 11:32:19.943409 13649 raft_consensus.cc:374] T 46cee4dad78541e4b18661ab0982800d P f5d0dd97f1b94d5e9b7fd8ccca5b8ee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:32:19.943464 13649 raft_consensus.cc:729] T 46cee4dad78541e4b18661ab0982800d P f5d0dd97f1b94d5e9b7fd8ccca5b8ee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5d0dd97f1b94d5e9b7fd8ccca5b8ee0, State: Initialized, Role: FOLLOWER
I20260926 11:32:19.943985 13649 consensus_queue.cc:260] T 46cee4dad78541e4b18661ab0982800d P f5d0dd97f1b94d5e9b7fd8ccca5b8ee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2ec2bd33ad480781c059a23173347d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42587 } } peers { permanent_uuid: "f5d0dd97f1b94d5e9b7fd8ccca5b8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40917 } } peers { permanent_uuid: "e97d17d5aeb14ef88b88b9db9d7d08b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45173 } }
I20260926 11:32:19.945140 13649 ts_tablet_manager.cc:1428] T 46cee4dad78541e4b18661ab0982800d P f5d0dd97f1b94d5e9b7fd8ccca5b8ee0: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 11:32:19.945739 13650 tablet_bootstrap.cc:492] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3: Bootstrap starting.
I20260926 11:32:19.948239 13650 tablet_bootstrap.cc:654] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3: Neither blocks nor log segments found. Creating new log.
I20260926 11:32:19.949119 13650 log.cc:824] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3: Log is configured to *not* fsync() on all Append() calls
I20260926 11:32:19.949981 13651 tablet_bootstrap.cc:492] T 46cee4dad78541e4b18661ab0982800d P 8f2ec2bd33ad480781c059a23173347d: Bootstrap starting.
I20260926 11:32:19.952265 13651 tablet_bootstrap.cc:654] T 46cee4dad78541e4b18661ab0982800d P 8f2ec2bd33ad480781c059a23173347d: Neither blocks nor log segments found. Creating new log.
I20260926 11:32:19.953114 13651 log.cc:824] T 46cee4dad78541e4b18661ab0982800d P 8f2ec2bd33ad480781c059a23173347d: Log is configured to *not* fsync() on all Append() calls
I20260926 11:32:19.953509 13346 heartbeater.cc:503] Master 127.0.98.62:33801 was elected leader, sending a full tablet report...
I20260926 11:32:19.957419 13650 tablet_bootstrap.cc:492] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3: No bootstrap required, opened a new log
I20260926 11:32:19.957648 13650 ts_tablet_manager.cc:1397] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260926 11:32:19.962214 13650 raft_consensus.cc:348] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2ec2bd33ad480781c059a23173347d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42587 } } peers { permanent_uuid: "f5d0dd97f1b94d5e9b7fd8ccca5b8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40917 } } peers { permanent_uuid: "e97d17d5aeb14ef88b88b9db9d7d08b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45173 } }
I20260926 11:32:19.962550 13650 raft_consensus.cc:374] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:32:19.962617 13650 raft_consensus.cc:729] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e97d17d5aeb14ef88b88b9db9d7d08b3, State: Initialized, Role: FOLLOWER
I20260926 11:32:19.965493 13651 tablet_bootstrap.cc:492] T 46cee4dad78541e4b18661ab0982800d P 8f2ec2bd33ad480781c059a23173347d: No bootstrap required, opened a new log
I20260926 11:32:19.965696 13651 ts_tablet_manager.cc:1397] T 46cee4dad78541e4b18661ab0982800d P 8f2ec2bd33ad480781c059a23173347d: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20260926 11:32:19.969786 13651 raft_consensus.cc:348] T 46cee4dad78541e4b18661ab0982800d P 8f2ec2bd33ad480781c059a23173347d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2ec2bd33ad480781c059a23173347d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42587 } } peers { permanent_uuid: "f5d0dd97f1b94d5e9b7fd8ccca5b8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40917 } } peers { permanent_uuid: "e97d17d5aeb14ef88b88b9db9d7d08b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45173 } }
I20260926 11:32:19.970031 13651 raft_consensus.cc:374] T 46cee4dad78541e4b18661ab0982800d P 8f2ec2bd33ad480781c059a23173347d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:32:19.970070 13651 raft_consensus.cc:729] T 46cee4dad78541e4b18661ab0982800d P 8f2ec2bd33ad480781c059a23173347d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f2ec2bd33ad480781c059a23173347d, State: Initialized, Role: FOLLOWER
I20260926 11:32:19.970556 13651 consensus_queue.cc:260] T 46cee4dad78541e4b18661ab0982800d P 8f2ec2bd33ad480781c059a23173347d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2ec2bd33ad480781c059a23173347d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42587 } } peers { permanent_uuid: "f5d0dd97f1b94d5e9b7fd8ccca5b8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40917 } } peers { permanent_uuid: "e97d17d5aeb14ef88b88b9db9d7d08b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45173 } }
I20260926 11:32:19.971750 13651 ts_tablet_manager.cc:1428] T 46cee4dad78541e4b18661ab0982800d P 8f2ec2bd33ad480781c059a23173347d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 11:32:19.972129 13480 heartbeater.cc:503] Master 127.0.98.62:33801 was elected leader, sending a full tablet report...
I20260926 11:32:19.983284 13650 consensus_queue.cc:260] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2ec2bd33ad480781c059a23173347d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42587 } } peers { permanent_uuid: "f5d0dd97f1b94d5e9b7fd8ccca5b8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40917 } } peers { permanent_uuid: "e97d17d5aeb14ef88b88b9db9d7d08b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45173 } }
I20260926 11:32:19.984170 13614 heartbeater.cc:503] Master 127.0.98.62:33801 was elected leader, sending a full tablet report...
I20260926 11:32:19.984367 13657 raft_consensus.cc:482] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:32:19.984738 13650 ts_tablet_manager.cc:1428] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3: Time spent starting tablet: real 0.027s user 0.008s sys 0.000s
I20260926 11:32:19.984573 13657 raft_consensus.cc:504] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2ec2bd33ad480781c059a23173347d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42587 } } peers { permanent_uuid: "f5d0dd97f1b94d5e9b7fd8ccca5b8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40917 } } peers { permanent_uuid: "e97d17d5aeb14ef88b88b9db9d7d08b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45173 } }
I20260926 11:32:19.990142 13657 leader_election.cc:288] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f2ec2bd33ad480781c059a23173347d (127.0.98.2:42587), f5d0dd97f1b94d5e9b7fd8ccca5b8ee0 (127.0.98.1:40917)
W20260926 11:32:19.992563 13347 tablet.cc:2369] T 46cee4dad78541e4b18661ab0982800d P f5d0dd97f1b94d5e9b7fd8ccca5b8ee0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:32:19.996493 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:19.989927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56171 (local address 127.0.98.1:40917)
0926 11:32:19.990053 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:19.990058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:19.990765 (+ 707us) server_negotiation.cc:407] Connection header received
0926 11:32:19.990942 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:19.990946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:19.990998 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:19.991045 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:32:19.991454 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:19.992213 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:19.993830 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:19.994493 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:19.994582 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:19.994689 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:19.996274 (+ 1585us) server_negotiation.cc:299] Negotiation successful
0926 11:32:19.996355 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 11:32:19.996619 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:19.989600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51949 (local address 127.0.98.2:42587)
0926 11:32:19.990107 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:19.990112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:19.990498 (+ 386us) server_negotiation.cc:407] Connection header received
0926 11:32:19.990606 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:19.990609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:19.990663 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:19.990741 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:32:19.991165 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:19.991995 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:19.994388 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:19.994972 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:19.995084 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:19.995329 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:19.996424 (+ 1095us) server_negotiation.cc:299] Negotiation successful
0926 11:32:19.996504 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409}
I20260926 11:32:19.996901 13660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:19.989810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40917 (local address 127.0.98.3:56171)
0926 11:32:19.990692 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:19.990704 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:19.990827 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:19.991177 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:19.991180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:19.991194 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:32:19.991327 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:19.991333 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:19.992354 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:19.992357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:19.993688 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:19.993699 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:19.995836 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:19.995839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:19.995934 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:19.996158 (+ 224us) client_negotiation.cc:769] Sending connection context
0926 11:32:19.996671 (+ 513us) client_negotiation.cc:240] Negotiation successful
0926 11:32:19.996745 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"mutex_wait_us":3,"thread_start_us":106,"threads_started":1}
I20260926 11:32:19.997473 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:19.989177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42587 (local address 127.0.98.3:51949)
0926 11:32:19.990387 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:19.990401 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:19.990514 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:19.990855 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:19.990858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:19.990877 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:32:19.991031 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:19.991039 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:19.992137 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:19.992140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:19.994236 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:19.994248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:19.996048 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:19.996051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:19.996289 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:19.996312 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 11:32:19.997255 (+ 943us) client_negotiation.cc:240] Negotiation successful
0926 11:32:19.997331 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1140,"mutex_wait_us":27,"thread_start_us":112,"threads_started":1}
I20260926 11:32:19.997478 13281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46cee4dad78541e4b18661ab0982800d" candidate_uuid: "e97d17d5aeb14ef88b88b9db9d7d08b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5d0dd97f1b94d5e9b7fd8ccca5b8ee0" is_pre_election: true
I20260926 11:32:19.997803 13281 raft_consensus.cc:2454] T 46cee4dad78541e4b18661ab0982800d P f5d0dd97f1b94d5e9b7fd8ccca5b8ee0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e97d17d5aeb14ef88b88b9db9d7d08b3 in term 0.
I20260926 11:32:19.998325 13503 leader_election.cc:302] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e97d17d5aeb14ef88b88b9db9d7d08b3, f5d0dd97f1b94d5e9b7fd8ccca5b8ee0; no voters:
I20260926 11:32:19.998600 13657 raft_consensus.cc:2781] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:32:19.998597 13419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46cee4dad78541e4b18661ab0982800d" candidate_uuid: "e97d17d5aeb14ef88b88b9db9d7d08b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2ec2bd33ad480781c059a23173347d" is_pre_election: true
I20260926 11:32:19.998690 13657 raft_consensus.cc:482] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:32:19.998735 13657 raft_consensus.cc:3037] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:32:19.998878 13419 raft_consensus.cc:2454] T 46cee4dad78541e4b18661ab0982800d P 8f2ec2bd33ad480781c059a23173347d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e97d17d5aeb14ef88b88b9db9d7d08b3 in term 0.
I20260926 11:32:20.000064 13657 raft_consensus.cc:504] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2ec2bd33ad480781c059a23173347d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42587 } } peers { permanent_uuid: "f5d0dd97f1b94d5e9b7fd8ccca5b8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40917 } } peers { permanent_uuid: "e97d17d5aeb14ef88b88b9db9d7d08b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45173 } }
I20260926 11:32:20.000711 13419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46cee4dad78541e4b18661ab0982800d" candidate_uuid: "e97d17d5aeb14ef88b88b9db9d7d08b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f2ec2bd33ad480781c059a23173347d"
I20260926 11:32:20.000823 13419 raft_consensus.cc:3037] T 46cee4dad78541e4b18661ab0982800d P 8f2ec2bd33ad480781c059a23173347d [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:32:20.002050 13281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46cee4dad78541e4b18661ab0982800d" candidate_uuid: "e97d17d5aeb14ef88b88b9db9d7d08b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5d0dd97f1b94d5e9b7fd8ccca5b8ee0"
I20260926 11:32:20.002170 13281 raft_consensus.cc:3037] T 46cee4dad78541e4b18661ab0982800d P f5d0dd97f1b94d5e9b7fd8ccca5b8ee0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:32:20.002197 13419 raft_consensus.cc:2454] T 46cee4dad78541e4b18661ab0982800d P 8f2ec2bd33ad480781c059a23173347d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e97d17d5aeb14ef88b88b9db9d7d08b3 in term 1.
I20260926 11:32:20.002544 13501 leader_election.cc:302] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f2ec2bd33ad480781c059a23173347d, e97d17d5aeb14ef88b88b9db9d7d08b3; no voters:
I20260926 11:32:20.003301 13659 raft_consensus.cc:2781] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:32:20.003391 13281 raft_consensus.cc:2454] T 46cee4dad78541e4b18661ab0982800d P f5d0dd97f1b94d5e9b7fd8ccca5b8ee0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e97d17d5aeb14ef88b88b9db9d7d08b3 in term 1.
I20260926 11:32:20.003517 13659 raft_consensus.cc:686] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3 [term 1 LEADER]: Becoming Leader. State: Replica: e97d17d5aeb14ef88b88b9db9d7d08b3, State: Running, Role: LEADER
I20260926 11:32:20.003898 13659 consensus_queue.cc:237] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2ec2bd33ad480781c059a23173347d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42587 } } peers { permanent_uuid: "f5d0dd97f1b94d5e9b7fd8ccca5b8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40917 } } peers { permanent_uuid: "e97d17d5aeb14ef88b88b9db9d7d08b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45173 } }
I20260926 11:32:20.006253 13657 leader_election.cc:288] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3 [CANDIDATE]: Term 1 election: Requested vote from peers 8f2ec2bd33ad480781c059a23173347d (127.0.98.2:42587), f5d0dd97f1b94d5e9b7fd8ccca5b8ee0 (127.0.98.1:40917)
I20260926 11:32:20.007433 13146 catalog_manager.cc:5495] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e97d17d5aeb14ef88b88b9db9d7d08b3 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e97d17d5aeb14ef88b88b9db9d7d08b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f2ec2bd33ad480781c059a23173347d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42587 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5d0dd97f1b94d5e9b7fd8ccca5b8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e97d17d5aeb14ef88b88b9db9d7d08b3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45173 } health_report { overall_health: HEALTHY } } }
I20260926 11:32:20.029569 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:20.024498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39152 (local address 127.0.98.3:45173)
0926 11:32:20.024610 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:20.024614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:20.024627 (+ 13us) server_negotiation.cc:407] Connection header received
0926 11:32:20.024725 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:20.024728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:20.024783 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:20.024919 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:32:20.025174 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:20.025951 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:20.027178 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:20.027514 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:20.027606 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:20.028028 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:32:20.028652 (+ 624us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:32:20.028764 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:20.029337 (+ 573us) server_negotiation.cc:299] Negotiation successful
0926 11:32:20.029398 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 11:32:20.036890 13281 raft_consensus.cc:1261] T 46cee4dad78541e4b18661ab0982800d P f5d0dd97f1b94d5e9b7fd8ccca5b8ee0 [term 1 FOLLOWER]: Refusing update from remote peer e97d17d5aeb14ef88b88b9db9d7d08b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:32:20.037693 13659 consensus_queue.cc:1035] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5d0dd97f1b94d5e9b7fd8ccca5b8ee0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 11:32:20.041352 13481 tablet.cc:2369] T 46cee4dad78541e4b18661ab0982800d P 8f2ec2bd33ad480781c059a23173347d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:32:20.046190 13419 raft_consensus.cc:1261] T 46cee4dad78541e4b18661ab0982800d P 8f2ec2bd33ad480781c059a23173347d [term 1 FOLLOWER]: Refusing update from remote peer e97d17d5aeb14ef88b88b9db9d7d08b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:32:20.046881 13659 consensus_queue.cc:1035] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f2ec2bd33ad480781c059a23173347d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:32:20.061720 13665 mvcc.cc:205] Tried to move back new op lower bound from 7333569904781910016 to 7333569904659693568. Current Snapshot: MvccSnapshot[applied={T|T < 7333569904781910016}]
I20260926 11:32:20.062201 13666 mvcc.cc:205] Tried to move back new op lower bound from 7333569904781910016 to 7333569904659693568. Current Snapshot: MvccSnapshot[applied={T|T < 7333569904781910016}]
I20260926 11:32:20.067632 13668 mvcc.cc:205] Tried to move back new op lower bound from 7333569904781910016 to 7333569904659693568. Current Snapshot: MvccSnapshot[applied={T|T < 7333569904781910016}]
I20260926 11:32:20.081442 13147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:32:20.081662 13147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:32:20.084379 13147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 272dc9d37b684ad7aa3fd9fca08ea68a: Sending DeleteTablet for 3 replicas of tablet 46cee4dad78541e4b18661ab0982800d
I20260926 11:32:20.085484 13269 tablet_service.cc:1514] Processing DeleteTablet for tablet 46cee4dad78541e4b18661ab0982800d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:32:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45478
I20260926 11:32:20.085505 13406 tablet_service.cc:1514] Processing DeleteTablet for tablet 46cee4dad78541e4b18661ab0982800d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:32:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37698
I20260926 11:32:20.086076 13548 tablet_service.cc:1514] Processing DeleteTablet for tablet 46cee4dad78541e4b18661ab0982800d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:32:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39138
I20260926 11:32:20.086647 13676 tablet_replica.cc:330] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3: stopping tablet replica
I20260926 11:32:20.087038 13676 raft_consensus.cc:2229] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:32:20.087168 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 13185
I20260926 11:32:20.088289 13676 raft_consensus.cc:2258] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:32:20.090166 13676 ts_tablet_manager.cc:1905] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:32:20.093010 13676 ts_tablet_manager.cc:1918] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:32:20.093173 13676 log.cc:1196] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790422180210734-392-0/minicluster-data/ts-2/wal/wals/46cee4dad78541e4b18661ab0982800d
I20260926 11:32:20.093576 13676 ts_tablet_manager.cc:1939] T 46cee4dad78541e4b18661ab0982800d P e97d17d5aeb14ef88b88b9db9d7d08b3: Deleting consensus metadata
I20260926 11:32:20.094949 13131 catalog_manager.cc:4841] TS e97d17d5aeb14ef88b88b9db9d7d08b3 (127.0.98.3:45173): tablet 46cee4dad78541e4b18661ab0982800d (table test-table [id=cc5c832cd4d14afaabffca817a1cc501]) successfully deleted
I20260926 11:32:20.096616 13674 tablet_replica.cc:330] T 46cee4dad78541e4b18661ab0982800d P 8f2ec2bd33ad480781c059a23173347d: stopping tablet replica
I20260926 11:32:20.097009 13674 raft_consensus.cc:2229] T 46cee4dad78541e4b18661ab0982800d P 8f2ec2bd33ad480781c059a23173347d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:32:20.097406 13674 raft_consensus.cc:2258] T 46cee4dad78541e4b18661ab0982800d P 8f2ec2bd33ad480781c059a23173347d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:32:20.101217 13674 ts_tablet_manager.cc:1905] T 46cee4dad78541e4b18661ab0982800d P 8f2ec2bd33ad480781c059a23173347d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 11:32:20.105443 13133 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40917 (error 108)
I20260926 11:32:20.107343 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 13350
I20260926 11:32:20.116158 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:20.115756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40917 (local address 127.0.0.1:45486)
0926 11:32:20.115967 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:20.116040 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260926 11:32:20.116549 13133 catalog_manager.cc:4584] TS f5d0dd97f1b94d5e9b7fd8ccca5b8ee0 (127.0.98.1:40917): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 46cee4dad78541e4b18661ab0982800d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40917: connect: Connection refused (error 111)
W20260926 11:32:20.118093 13643 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:32:20.117419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42587 (local address 127.0.0.1:37714)
0926 11:32:20.117569 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:20.117584 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:20.117677 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:20.118002 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42587: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 11:32:20.118742 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 13482
I20260926 11:32:20.128547 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 13114
I20260926 11:32:20.136540 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 11:32:20.481455 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 11:32:20.481563 392 mini_ranger.cc:66] Stopping Ranger...
I20260926 11:32:20.826426 392 mini_ranger.cc:68] Stopped Ranger
2026-09-26 11:32:20.826 UTC [12181] LOG: received smart shutdown request
2026-09-26 11:32:20.830 UTC [12181] LOG: background worker "logical replication launcher" (PID 12190) exited with exit code 1
2026-09-26 11:32:20.830 UTC [12185] LOG: shutting down
2026-09-26 11:32:21.000 UTC [12181] LOG: database system is shut down
2026-09-26T11:32:21Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49576 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:32:21 dist-test-slave-mrck krb5kdc[13686](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:32:21 dist-test-slave-mrck krb5kdc[13686](info): set up 2 sockets
Sep 26 11:32:21 dist-test-slave-mrck krb5kdc[13686](info): commencing operation
krb5kdc: starting...
W20260926 11:32:22.175911 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:32:22 dist-test-slave-mrck krb5kdc[13686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:32:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:32:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:32:22.298761 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36465
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36465,127.0.98.61:36667,127.0.98.60:34547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:32:22.407644 13699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:32:22.407970 13699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:32:22.408051 13699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:32:22.412032 13699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:32:22.412119 13699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:32:22.412156 13699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:32:22.412210 13699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:32:22.412254 13699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:32:22.417507 13699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36465,127.0.98.61:36667,127.0.98.60:34547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:32:22.418572 13699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:32:22.419574 13699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:32:22.427693 13705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:22.427839 13699 server_base.cc:1034] running on GCE node
W20260926 11:32:22.427778 13707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:22.427695 13704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:22.428506 13699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:32:22.429539 13699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:32:22.430702 13699 hybrid_clock.cc:648] HybridClock initialized: now 1790422342430684 us; error 30 us; skew 500 ppm
Sep 26 11:32:22 dist-test-slave-mrck krb5kdc[13686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:32:22.433439 13699 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:32:22.434388 13699 webserver.cc:460] Webserver started at http://127.0.98.62:40775/ using document root <none> and password file <none>
I20260926 11:32:22.434929 13699 fs_manager.cc:359] Metadata directory not provided
I20260926 11:32:22.435016 13699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:32:22.435225 13699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:32:22.437250 13699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "507fcc3e67694e4ab55eb46004aca4ff"
format_stamp: "Formatted at 2026-09-26 11:32:22 on dist-test-slave-mrck"
server_key: "059461e9b3b1140ea47c9ffaf419a38e"
server_key_iv: "5161d3810b90cf2f5da9b97c3b697717"
server_key_version: "encryptionkey@0"
I20260926 11:32:22.437808 13699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "507fcc3e67694e4ab55eb46004aca4ff"
format_stamp: "Formatted at 2026-09-26 11:32:22 on dist-test-slave-mrck"
server_key: "059461e9b3b1140ea47c9ffaf419a38e"
server_key_iv: "5161d3810b90cf2f5da9b97c3b697717"
server_key_version: "encryptionkey@0"
I20260926 11:32:22.441339 13699 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 11:32:22.443674 13714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:22.444892 13699 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:32:22.445036 13699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "507fcc3e67694e4ab55eb46004aca4ff"
format_stamp: "Formatted at 2026-09-26 11:32:22 on dist-test-slave-mrck"
server_key: "059461e9b3b1140ea47c9ffaf419a38e"
server_key_iv: "5161d3810b90cf2f5da9b97c3b697717"
server_key_version: "encryptionkey@0"
I20260926 11:32:22.445147 13699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:32:22.466506 13699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:32:22.469548 13699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:32:22.469767 13699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:32:22.477217 13699 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36465
I20260926 11:32:22.477242 13766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36465 every 8 connection(s)
I20260926 11:32:22.478153 13699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:32:22.480865 13767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:32:22.484793 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 13699
I20260926 11:32:22.484892 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:32:22.484656 13767 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36465 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:32:22.485193 392 external_mini_cluster.cc:1428] Setting key 2fbe4bc3999b3e248e56b5d0de3389a4
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 26 11:32:22 dist-test-slave-mrck krb5kdc[13686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:32:22.501542 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.487002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41956 (local address 127.0.98.62:36465)
0926 11:32:22.487564 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:22.487576 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:22.487680 (+ 104us) server_negotiation.cc:407] Connection header received
0926 11:32:22.488758 (+ 1078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:22.488767 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:22.489023 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:22.489248 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:22.490017 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.491510 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.492580 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.493194 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.493382 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:22.496690 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:22.496700 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:22.496717 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:22.496749 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:22.498373 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:22.498945 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:22.498950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:22.498951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:22.499047 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:22.499404 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:22.499407 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:22.499408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:22.499845 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:22.499943 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:22.500410 (+ 467us) server_negotiation.cc:299] Negotiation successful
0926 11:32:22.500616 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"spinlock_wait_cycles":18304,"thread_start_us":176,"threads_started":1}
I20260926 11:32:22.501544 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.487001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36465 (local address 127.0.0.1:41956)
0926 11:32:22.487500 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:22.487554 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:22.488426 (+ 872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:22.489267 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:22.489272 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:22.489620 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:32:22.489864 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:22.489876 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.491657 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.491660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:22.492454 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:22.492464 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.493373 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.493379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:22.493485 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:22.494089 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:32:22.494104 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:32:22.496535 (+ 2431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:32:22.498485 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:22.498496 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:22.498510 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:22.498832 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:22.499140 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:22.499146 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:22.499151 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:22.499301 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:22.499956 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:32:22.499966 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:32:22.500141 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 11:32:22.500342 (+ 201us) client_negotiation.cc:240] Negotiation successful
0926 11:32:22.500722 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":33,"thread_start_us":153,"threads_started":1}
I20260926 11:32:22.504287 13767 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36667 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:32:22.505039 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.504796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36667 (local address 127.0.0.1:56710)
0926 11:32:22.504909 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:22.504978 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 11:32:22.505260 13716 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36667: connect: Connection refused (error 111)
I20260926 11:32:22.506110 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.505900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36667 (local address 127.0.0.1:56720)
0926 11:32:22.506003 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:22.506058 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 11:32:22.506276 13767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36667: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36667: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 11:32:22.511127 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36667
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36465,127.0.98.61:36667,127.0.98.60:34547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:32:22.555516 13767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36667 } attempt: 1
I20260926 11:32:22.556452 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.556099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36667 (local address 127.0.0.1:56730)
0926 11:32:22.556287 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:22.556363 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 11:32:22.557188 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.556964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36667 (local address 127.0.0.1:56746)
0926 11:32:22.557069 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:22.557121 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 11:32:22.557343 13767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36667: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36667: connect: Connection refused (error 111)
W20260926 11:32:22.622442 13774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:32:22.622711 13774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:32:22.622753 13774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 11:32:22.625525 13767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36667 } attempt: 2
I20260926 11:32:22.626413 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.626090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36667 (local address 127.0.0.1:56758)
0926 11:32:22.626257 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:22.626326 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 11:32:22.627096 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.626896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36667 (local address 127.0.0.1:56768)
0926 11:32:22.626990 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:22.627037 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 11:32:22.627260 13774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:32:22.627247 13767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36667: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36667: connect: Connection refused (error 111)
W20260926 11:32:22.627341 13774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:32:22.627377 13774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:32:22.627413 13774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:32:22.627445 13774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:32:22.632831 13774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36465,127.0.98.61:36667,127.0.98.60:34547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:32:22.633903 13774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:32:22.634920 13774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:32:22.643443 13780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:22.643544 13774 server_base.cc:1034] running on GCE node
W20260926 11:32:22.643520 13779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:22.643594 13782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:22.644213 13774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:32:22.645303 13774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:32:22.646462 13774 hybrid_clock.cc:648] HybridClock initialized: now 1790422342646445 us; error 32 us; skew 500 ppm
Sep 26 11:32:22 dist-test-slave-mrck krb5kdc[13686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:32:22.649261 13774 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 11:32:22.650240 13774 webserver.cc:460] Webserver started at http://127.0.98.61:40861/ using document root <none> and password file <none>
I20260926 11:32:22.650785 13774 fs_manager.cc:359] Metadata directory not provided
I20260926 11:32:22.650843 13774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:32:22.651080 13774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:32:22.653146 13774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-1/data/instance:
uuid: "c18bfc13bee2485c8e2e47ee0582e496"
format_stamp: "Formatted at 2026-09-26 11:32:22 on dist-test-slave-mrck"
server_key: "96b0a70abcc064b06da295f7008b15ac"
server_key_iv: "d4b48c78f8697b8a3458a22b993f44ec"
server_key_version: "encryptionkey@0"
I20260926 11:32:22.653692 13774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-1/wal/instance:
uuid: "c18bfc13bee2485c8e2e47ee0582e496"
format_stamp: "Formatted at 2026-09-26 11:32:22 on dist-test-slave-mrck"
server_key: "96b0a70abcc064b06da295f7008b15ac"
server_key_iv: "d4b48c78f8697b8a3458a22b993f44ec"
server_key_version: "encryptionkey@0"
I20260926 11:32:22.657387 13774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 11:32:22.659943 13789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:22.661163 13774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 11:32:22.661283 13774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-1/wal
uuid: "c18bfc13bee2485c8e2e47ee0582e496"
format_stamp: "Formatted at 2026-09-26 11:32:22 on dist-test-slave-mrck"
server_key: "96b0a70abcc064b06da295f7008b15ac"
server_key_iv: "d4b48c78f8697b8a3458a22b993f44ec"
server_key_version: "encryptionkey@0"
I20260926 11:32:22.661396 13774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:32:22.671424 13774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:32:22.674340 13774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:32:22.674540 13774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:32:22.681790 13774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36667
I20260926 11:32:22.681813 13841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36667 every 8 connection(s)
I20260926 11:32:22.682776 13774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-1/data/info.pb
I20260926 11:32:22.685510 13842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:32:22.686992 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 13774
I20260926 11:32:22.687143 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-1/wal/instance
I20260926 11:32:22.687428 392 external_mini_cluster.cc:1428] Setting key bc9a8d2096ea4e9a4788bfdd2aa13f86
I20260926 11:32:22.689410 13842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36465 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 26 11:32:22 dist-test-slave-mrck krb5kdc[13686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:32:22.703074 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.691344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41958 (local address 127.0.98.62:36465)
0926 11:32:22.691474 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:22.691478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:22.692552 (+ 1074us) server_negotiation.cc:407] Connection header received
0926 11:32:22.693455 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:22.693460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:22.693521 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:22.693630 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:22.694739 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.695281 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.696471 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.696722 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.696833 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:22.699382 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:22.699385 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:22.699387 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:22.699411 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:22.700952 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:22.701457 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:22.701463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:22.701464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:22.701604 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:22.701938 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:22.701942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:22.701944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:22.702132 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:22.702238 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:22.702765 (+ 527us) server_negotiation.cc:299] Negotiation successful
0926 11:32:22.702889 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 11:32:22.703943 13845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.691882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36465 (local address 127.0.0.1:41958)
0926 11:32:22.692385 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:22.692433 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:22.693240 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:22.693793 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:22.693800 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:22.694118 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:32:22.694595 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:22.694607 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.695405 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.695415 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:22.696340 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:22.696349 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.696870 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.696875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:22.697040 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:22.697671 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:32:22.697688 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:32:22.699213 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:32:22.701079 (+ 1866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:22.701085 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:22.701099 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:22.701353 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:22.701693 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:22.701696 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:22.701698 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:22.701839 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:22.702237 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:32:22.702245 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:32:22.702498 (+ 253us) client_negotiation.cc:769] Sending connection context
0926 11:32:22.702699 (+ 201us) client_negotiation.cc:240] Negotiation successful
0926 11:32:22.703021 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":151,"threads_started":1}
I20260926 11:32:22.705497 13842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36667 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 11:32:22 dist-test-slave-mrck krb5kdc[13686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 11:32:22.712633 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34547
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36465,127.0.98.61:36667,127.0.98.60:34547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:32:22.718385 13845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.706013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36667 (local address 127.0.0.1:56782)
0926 11:32:22.706195 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:22.706210 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:22.706288 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:22.706850 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:22.706852 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:22.706983 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:32:22.707072 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:22.707077 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.708185 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.708188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:22.708828 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:22.708837 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.709724 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.709732 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:22.709802 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:22.710281 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:32:22.710292 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:32:22.711502 (+ 1210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:32:22.716566 (+ 5064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:22.716571 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:22.716575 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:22.716841 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:22.717197 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:22.717200 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:22.717201 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:22.717278 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:22.717835 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:32:22.717840 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:32:22.717944 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 11:32:22.718074 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 11:32:22.718212 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260926 11:32:22.718415 13767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36667 } attempt: 3
I20260926 11:32:22.719182 13847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.706091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56782 (local address 127.0.98.61:36667)
0926 11:32:22.706376 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:22.706394 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:22.706419 (+ 25us) server_negotiation.cc:407] Connection header received
0926 11:32:22.706496 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:22.706503 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:22.706714 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:22.706863 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:22.707223 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.708092 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.708931 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.709605 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.709693 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:22.711646 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:22.711654 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:22.711665 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:22.711684 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:22.716423 (+ 4739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:22.716964 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:22.716970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:22.716972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:22.717091 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:22.717364 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:22.717369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:22.717371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:22.717714 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:22.717846 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:22.718212 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 11:32:22.718342 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":110,"threads_started":1}
I20260926 11:32:22.721489 13842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34547 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:32:22.723721 13845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.722482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34547 (local address 127.0.0.1:47686)
0926 11:32:22.723564 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:22.723646 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1004}
W20260926 11:32:22.723968 13790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34547: connect: Connection refused (error 111)
I20260926 11:32:22.725080 13851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.724673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34547 (local address 127.0.0.1:47700)
0926 11:32:22.724918 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:22.724984 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":92,"threads_started":1}
W20260926 11:32:22.725339 13842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34547: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34547: connect: Connection refused (error 111)
Sep 26 11:32:22 dist-test-slave-mrck krb5kdc[13686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 11:32:22.730901 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.718860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36667 (local address 127.0.0.1:56788)
0926 11:32:22.718968 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:22.718984 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:22.719073 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:22.721341 (+ 2268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:22.721347 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:22.721590 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:32:22.722215 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:22.722230 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.723489 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.723493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:22.724328 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:22.724340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.725494 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.725498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:22.725627 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:22.726097 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:32:22.726111 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:32:22.727485 (+ 1374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:32:22.729383 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:22.729386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:22.729387 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:22.729579 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:22.729836 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:22.729838 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:22.729839 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:22.729910 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:22.730462 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:32:22.730464 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:32:22.730556 (+ 92us) client_negotiation.cc:769] Sending connection context
0926 11:32:22.730653 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 11:32:22.730774 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 11:32:22.730997 13847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.720384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56788 (local address 127.0.98.61:36667)
0926 11:32:22.721082 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:22.721088 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:22.721102 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:32:22.721138 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:22.721141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:22.721195 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:22.722608 (+ 1413us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:22.722681 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.723360 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.724470 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.725359 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.725467 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:22.727617 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:22.727621 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:22.727624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:22.727650 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:22.729259 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:22.729665 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:22.729669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:22.729671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:22.729752 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:22.729992 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:22.729995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:22.729997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:22.730203 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:22.730296 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:22.730720 (+ 424us) server_negotiation.cc:299] Negotiation successful
0926 11:32:22.730832 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":116,"threads_started":1}
I20260926 11:32:22.731791 13767 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34547 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:32:22.732470 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.732248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34547 (local address 127.0.0.1:47714)
0926 11:32:22.732360 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:22.732416 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 11:32:22.733186 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.732966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34547 (local address 127.0.0.1:47726)
0926 11:32:22.733101 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:22.733141 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260926 11:32:22.733349 13767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34547: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34547: connect: Connection refused (error 111)
I20260926 11:32:22.764537 13767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34547 } attempt: 1
I20260926 11:32:22.765398 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.765109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34547 (local address 127.0.0.1:47730)
0926 11:32:22.765254 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:22.765320 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 11:32:22.766213 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.765982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34547 (local address 127.0.0.1:47746)
0926 11:32:22.766111 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:22.766164 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 11:32:22.766364 13767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34547: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34547: connect: Connection refused (error 111)
I20260926 11:32:22.774616 13842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34547 } attempt: 1
I20260926 11:32:22.775487 13851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.775208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34547 (local address 127.0.0.1:47750)
0926 11:32:22.775355 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:22.775415 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 11:32:22.776278 13851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.776098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34547 (local address 127.0.0.1:47756)
0926 11:32:22.776191 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:22.776232 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 11:32:22.776472 13842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34547: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34547: connect: Connection refused (error 111)
W20260926 11:32:22.832003 13848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:32:22.832288 13848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:32:22.832381 13848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:32:22.836542 13848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:32:22.836624 13848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:32:22.836648 13848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:32:22.836669 13848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:32:22.836742 13848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:32:22.841576 13767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34547 } attempt: 2
I20260926 11:32:22.842556 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.842206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34547 (local address 127.0.0.1:47760)
0926 11:32:22.842386 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:22.842475 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 11:32:22.842540 13848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36465,127.0.98.61:36667,127.0.98.60:34547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:32:22.843396 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.843125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34547 (local address 127.0.0.1:47764)
0926 11:32:22.843259 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:22.843322 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 11:32:22.843591 13767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34547: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34547: connect: Connection refused (error 111)
I20260926 11:32:22.843971 13848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:32:22.844630 13842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34547 } attempt: 2
I20260926 11:32:22.845029 13848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 11:32:22.845265 13851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.845030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34547 (local address 127.0.0.1:47768)
0926 11:32:22.845146 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:22.845206 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 11:32:22.845894 13851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.845699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34547 (local address 127.0.0.1:47780)
0926 11:32:22.845799 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:22.845843 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 11:32:22.846019 13842 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34547: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34547: connect: Connection refused (error 111)
W20260926 11:32:22.853869 13859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:22.853924 13856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:22.854034 13848 server_base.cc:1034] running on GCE node
W20260926 11:32:22.854038 13857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:22.854812 13848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:32:22.856019 13848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:32:22.857227 13848 hybrid_clock.cc:648] HybridClock initialized: now 1790422342857212 us; error 24 us; skew 500 ppm
Sep 26 11:32:22 dist-test-slave-mrck krb5kdc[13686](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:32:22.860018 13848 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 11:32:22.860976 13848 webserver.cc:460] Webserver started at http://127.0.98.60:44855/ using document root <none> and password file <none>
I20260926 11:32:22.861557 13848 fs_manager.cc:359] Metadata directory not provided
I20260926 11:32:22.861654 13848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:32:22.861897 13848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:32:22.864027 13848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-2/data/instance:
uuid: "aa065c72738a4c95ac402c16738b1574"
format_stamp: "Formatted at 2026-09-26 11:32:22 on dist-test-slave-mrck"
server_key: "4d1c4789b614726ec87398c933bc3e2a"
server_key_iv: "89083c4d97c8b12aba731366114c4973"
server_key_version: "encryptionkey@0"
I20260926 11:32:22.864631 13848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-2/wal/instance:
uuid: "aa065c72738a4c95ac402c16738b1574"
format_stamp: "Formatted at 2026-09-26 11:32:22 on dist-test-slave-mrck"
server_key: "4d1c4789b614726ec87398c933bc3e2a"
server_key_iv: "89083c4d97c8b12aba731366114c4973"
server_key_version: "encryptionkey@0"
I20260926 11:32:22.868664 13848 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 11:32:22.871206 13866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:22.872556 13848 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 11:32:22.872709 13848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-2/wal
uuid: "aa065c72738a4c95ac402c16738b1574"
format_stamp: "Formatted at 2026-09-26 11:32:22 on dist-test-slave-mrck"
server_key: "4d1c4789b614726ec87398c933bc3e2a"
server_key_iv: "89083c4d97c8b12aba731366114c4973"
server_key_version: "encryptionkey@0"
I20260926 11:32:22.872828 13848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:32:22.898972 13848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:32:22.902062 13848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:32:22.902305 13848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:32:22.909934 13848 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34547
I20260926 11:32:22.909986 13918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34547 every 8 connection(s)
I20260926 11:32:22.910959 13848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-2/data/info.pb
I20260926 11:32:22.913841 13919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:32:22.917570 13919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36465 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:32:22.919632 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 13848
I20260926 11:32:22.919795 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790422180210734-392-0/minicluster-data/master-2/wal/instance
I20260926 11:32:22.920094 392 external_mini_cluster.cc:1428] Setting key 67366da39c3e5844e259b2e319961400
Sep 26 11:32:22 dist-test-slave-mrck krb5kdc[13686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 11:32:22 dist-test-slave-mrck krb5kdc[13686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 11:32:22.932253 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.919469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41960 (local address 127.0.98.62:36465)
0926 11:32:22.919644 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:22.919649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:22.920538 (+ 889us) server_negotiation.cc:407] Connection header received
0926 11:32:22.921702 (+ 1164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:22.921706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:22.921755 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:22.921864 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:22.923035 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.923834 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.925242 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.925577 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.925689 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:22.928371 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:22.928375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:22.928378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:22.928409 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:22.929904 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:22.930436 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:22.930440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:22.930442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:22.930527 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:22.930845 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:22.930849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:22.930851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:22.931104 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:22.931176 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:22.931928 (+ 752us) server_negotiation.cc:299] Negotiation successful
0926 11:32:22.932063 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 11:32:22.933116 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.919891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36465 (local address 127.0.0.1:41960)
0926 11:32:22.920364 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:22.920415 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:22.921497 (+ 1082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:22.921975 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:22.921982 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:22.922514 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:32:22.922886 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:22.922899 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.923948 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.923951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:22.925112 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:22.925123 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.925728 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.925731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:22.925914 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:22.926532 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:32:22.926547 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:32:22.928221 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:32:22.930002 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:22.930017 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:22.930026 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:22.930340 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:22.930621 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:22.930624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:22.930625 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:22.930753 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:22.931207 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:32:22.931215 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:32:22.931604 (+ 389us) client_negotiation.cc:769] Sending connection context
0926 11:32:22.931849 (+ 245us) client_negotiation.cc:240] Negotiation successful
0926 11:32:22.932107 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":130,"threads_started":1}
I20260926 11:32:22.933125 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.920761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47784 (local address 127.0.98.60:34547)
0926 11:32:22.921100 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:22.921109 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:22.921141 (+ 32us) server_negotiation.cc:407] Connection header received
0926 11:32:22.921493 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:22.921509 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:22.921748 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:22.921928 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:22.922879 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.924138 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.925241 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.925851 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.925967 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:22.928904 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:22.928918 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:22.928930 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:22.928950 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:22.930921 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:22.931373 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:22.931378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:22.931379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:22.931456 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:22.931787 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:22.931790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:22.931792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:22.932086 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:22.932275 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:22.932619 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 11:32:22.932777 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":118,"threads_started":1}
I20260926 11:32:22.934333 13919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36667 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:32:22.937172 13842 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34547 } attempt: 3
Sep 26 11:32:22 dist-test-slave-mrck krb5kdc[13686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 11:32:22 dist-test-slave-mrck krb5kdc[13686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:32:22.942776 13767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34547 } attempt: 3
I20260926 11:32:22.943120 13847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.934938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56802 (local address 127.0.98.61:36667)
0926 11:32:22.935079 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:22.935085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:22.935103 (+ 18us) server_negotiation.cc:407] Connection header received
0926 11:32:22.935147 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:22.935150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:22.935220 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:22.935300 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:22.935719 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.936279 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.937249 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.937583 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.937687 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:22.939680 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:22.939684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:22.939687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:22.939713 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:22.941259 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:22.941700 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:22.941703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:22.941705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:22.941788 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:22.942103 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:22.942108 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:22.942110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:22.942338 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:22.942445 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:22.942810 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 11:32:22.942952 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 11:32:22 dist-test-slave-mrck krb5kdc[13686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 11:32:22.943578 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.934834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36667 (local address 127.0.0.1:56802)
0926 11:32:22.934955 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:22.934985 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:22.935063 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:22.935323 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:22.935326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:22.935492 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:32:22.935617 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:22.935625 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.936385 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.936392 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:22.937099 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:22.937108 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.937693 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.937696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:22.937801 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:22.938238 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:32:22.938252 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:32:22.939546 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:32:22.941366 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:22.941371 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:22.941373 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:22.941571 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:22.941905 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:22.941910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:22.941912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:22.941996 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:22.942493 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:32:22.942499 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:32:22.942592 (+ 93us) client_negotiation.cc:769] Sending connection context
0926 11:32:22.943252 (+ 660us) client_negotiation.cc:240] Negotiation successful
0926 11:32:22.943378 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 11:32:22.944276 13919 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34547 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:32:22.946170 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.935709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41964 (local address 127.0.98.62:36465)
0926 11:32:22.935853 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:22.935858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:22.935896 (+ 38us) server_negotiation.cc:407] Connection header received
0926 11:32:22.936020 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:22.936023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:22.936061 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:22.936149 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:22.936826 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.937375 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.938373 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.938702 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.938796 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:22.941571 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:22.941577 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:22.941581 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:22.941611 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:22.943368 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:22.944708 (+ 1340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:22.944713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:22.944718 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:22.944809 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:22.945128 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:22.945131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:22.945133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:22.945357 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:22.945555 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:22.945906 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 11:32:22.946026 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 11:32:22.948226 13851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.937519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34547 (local address 127.0.0.1:47798)
0926 11:32:22.937733 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:22.937746 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:22.937813 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:22.938502 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:22.938505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:22.938701 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:32:22.938845 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:22.938852 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.939979 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.939986 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:22.941000 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:22.941014 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.941672 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.941679 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:22.941797 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:22.942326 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:32:22.942338 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:32:22.944148 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:32:22.946606 (+ 2458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:22.946609 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:22.946611 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:22.946842 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:22.947177 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:22.947180 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:22.947181 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:22.947267 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:22.947686 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:32:22.947689 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:32:22.947817 (+ 128us) client_negotiation.cc:769] Sending connection context
0926 11:32:22.947913 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 11:32:22.948057 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"spinlock_wait_cycles":3840}
I20260926 11:32:22.948776 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.937924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47798 (local address 127.0.98.60:34547)
0926 11:32:22.938264 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:22.938267 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:22.938277 (+ 10us) server_negotiation.cc:407] Connection header received
0926 11:32:22.938304 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:22.938306 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:22.938366 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:22.938953 (+ 587us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:22.939031 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.939837 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.941149 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.941538 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.941640 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:22.944278 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:22.944281 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:22.944285 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:22.944312 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:22.946487 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:22.946968 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:22.946974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:22.946979 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:22.947058 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:22.947377 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:22.947380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:22.947381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:22.947578 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:22.948377 (+ 799us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:22.948492 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 11:32:22.948594 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280}
Sep 26 11:32:22 dist-test-slave-mrck krb5kdc[13686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 11:32:22 dist-test-slave-mrck krb5kdc[13686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 11:32:22.952574 13842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496: Bootstrap starting.
I20260926 11:32:22.953367 13921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.944710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34547 (local address 127.0.0.1:47816)
0926 11:32:22.944819 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:22.944832 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:22.944910 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:22.945398 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:22.945401 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:22.945571 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:32:22.945686 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:22.945693 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.947038 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.947041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:22.947847 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:22.947856 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.948503 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.948506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:22.948583 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:22.948975 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:32:22.948987 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:32:22.950323 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:32:22.951957 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:22.951960 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:22.951962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:22.952180 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:22.952419 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:22.952421 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:22.952423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:22.952488 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:22.952929 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:32:22.952931 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:32:22.953002 (+ 71us) client_negotiation.cc:769] Sending connection context
0926 11:32:22.953085 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 11:32:22.953198 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":3}
I20260926 11:32:22.954205 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.944941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47816 (local address 127.0.98.60:34547)
0926 11:32:22.945197 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:22.945205 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:22.945219 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:32:22.945255 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:22.945261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:22.945313 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:22.945398 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:22.946127 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.946936 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.947956 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.948295 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.948408 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:22.950469 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:22.950472 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:22.950475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:22.950498 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:22.951841 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:22.952272 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:22.952275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:22.952276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:22.952341 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:22.952560 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:22.952563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:22.952564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:22.952809 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:22.952900 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:22.953558 (+ 658us) server_negotiation.cc:299] Negotiation successful
0926 11:32:22.954004 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":151,"server-negotiator.queue_time_us":194,"thread_start_us":93,"threads_started":1}
I20260926 11:32:22.955688 13842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496: Neither blocks nor log segments found. Creating new log.
Sep 26 11:32:22 dist-test-slave-mrck krb5kdc[13686](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 11:32:22.956545 13770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.943140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34547 (local address 127.0.0.1:47810)
0926 11:32:22.943368 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:22.943382 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:22.943463 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:22.944813 (+ 1350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:22.944815 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:22.944996 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:32:22.945117 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:22.945123 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.946742 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.946745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:22.947496 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:22.947505 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.948191 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.948195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:22.948301 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:22.949963 (+ 1662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:32:22.949978 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:32:22.951884 (+ 1906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:32:22.954812 (+ 2928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:22.954815 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:22.954817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:22.955052 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:22.955348 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:22.955352 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:22.955353 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:22.955435 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:22.956049 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:32:22.956054 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:32:22.956150 (+ 96us) client_negotiation.cc:769] Sending connection context
0926 11:32:22.956225 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 11:32:22.956358 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260926 11:32:22.956710 13842 log.cc:824] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496: Log is configured to *not* fsync() on all Append() calls
I20260926 11:32:22.957338 13919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa065c72738a4c95ac402c16738b1574: Bootstrap starting.
I20260926 11:32:22.957911 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.943891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47810 (local address 127.0.98.60:34547)
0926 11:32:22.944243 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:22.944252 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:22.944265 (+ 13us) server_negotiation.cc:407] Connection header received
0926 11:32:22.944299 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:22.944304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:22.944344 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:22.945216 (+ 872us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:22.945562 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.946626 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.947638 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.948069 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.952012 (+ 3943us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:22.952108 (+ 96us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:22.952112 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:22.952114 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:22.952132 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:22.954691 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:22.955174 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:22.955177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:22.955179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:22.955252 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:22.955561 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:22.955567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:22.955572 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:22.955861 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:22.957589 (+ 1728us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:22.957664 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 11:32:22.957764 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":276,"thread_start_us":157,"threads_started":1}
I20260926 11:32:22.958911 13842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496: No bootstrap required, opened a new log
I20260926 11:32:22.960034 13847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.947743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56810 (local address 127.0.98.61:36667)
0926 11:32:22.947837 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:22.947841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:22.948042 (+ 201us) server_negotiation.cc:407] Connection header received
0926 11:32:22.948259 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:22.948262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:22.948304 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:22.948351 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:22.949352 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.950150 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.952805 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.953136 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.953206 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:22.956463 (+ 3257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:22.956466 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:22.956486 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:22.956510 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:22.958086 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:22.958605 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:22.958609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:22.958611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:22.958696 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:22.959005 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:22.959008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:22.959010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:22.959200 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:22.959361 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:22.959717 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 11:32:22.959851 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 11:32:22.960040 13919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa065c72738a4c95ac402c16738b1574: Neither blocks nor log segments found. Creating new log.
I20260926 11:32:22.960750 13767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 507fcc3e67694e4ab55eb46004aca4ff: Bootstrap starting.
I20260926 11:32:22.960870 13919 log.cc:824] T 00000000000000000000000000000000 P aa065c72738a4c95ac402c16738b1574: Log is configured to *not* fsync() on all Append() calls
I20260926 11:32:22.962922 13919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa065c72738a4c95ac402c16738b1574: No bootstrap required, opened a new log
I20260926 11:32:22.963212 13767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 507fcc3e67694e4ab55eb46004aca4ff: Neither blocks nor log segments found. Creating new log.
I20260926 11:32:22.963146 13842 raft_consensus.cc:348] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507fcc3e67694e4ab55eb46004aca4ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36465 } } peers { permanent_uuid: "c18bfc13bee2485c8e2e47ee0582e496" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36667 } } peers { permanent_uuid: "aa065c72738a4c95ac402c16738b1574" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34547 } }
I20260926 11:32:22.963375 13842 raft_consensus.cc:374] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:32:22.963430 13842 raft_consensus.cc:729] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18bfc13bee2485c8e2e47ee0582e496, State: Initialized, Role: FOLLOWER
I20260926 11:32:22.963529 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:32:22.963927 13842 consensus_queue.cc:260] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507fcc3e67694e4ab55eb46004aca4ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36465 } } peers { permanent_uuid: "c18bfc13bee2485c8e2e47ee0582e496" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36667 } } peers { permanent_uuid: "aa065c72738a4c95ac402c16738b1574" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34547 } }
I20260926 11:32:22.964033 13767 log.cc:824] T 00000000000000000000000000000000 P 507fcc3e67694e4ab55eb46004aca4ff: Log is configured to *not* fsync() on all Append() calls
I20260926 11:32:22.965339 13842 sys_catalog.cc:564] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:32:22.967901 13919 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa065c72738a4c95ac402c16738b1574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507fcc3e67694e4ab55eb46004aca4ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36465 } } peers { permanent_uuid: "c18bfc13bee2485c8e2e47ee0582e496" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36667 } } peers { permanent_uuid: "aa065c72738a4c95ac402c16738b1574" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34547 } }
I20260926 11:32:22.968128 13919 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa065c72738a4c95ac402c16738b1574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:32:22.968181 13919 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa065c72738a4c95ac402c16738b1574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa065c72738a4c95ac402c16738b1574, State: Initialized, Role: FOLLOWER
I20260926 11:32:22.968658 13919 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa065c72738a4c95ac402c16738b1574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507fcc3e67694e4ab55eb46004aca4ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36465 } } peers { permanent_uuid: "c18bfc13bee2485c8e2e47ee0582e496" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36667 } } peers { permanent_uuid: "aa065c72738a4c95ac402c16738b1574" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34547 } }
I20260926 11:32:22.968953 13928 sys_catalog.cc:455] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507fcc3e67694e4ab55eb46004aca4ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36465 } } peers { permanent_uuid: "c18bfc13bee2485c8e2e47ee0582e496" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36667 } } peers { permanent_uuid: "aa065c72738a4c95ac402c16738b1574" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34547 } } }
I20260926 11:32:22.969055 13928 sys_catalog.cc:458] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:32:22.969905 13919 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa065c72738a4c95ac402c16738b1574 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:32:22.970258 13767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 507fcc3e67694e4ab55eb46004aca4ff: No bootstrap required, opened a new log
W20260926 11:32:22.970778 13947 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:32:22.970849 13947 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260926 11:32:22.974483 13960 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aa065c72738a4c95ac402c16738b1574: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:32:22.974560 13960 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:32:22.974025 13848 master_runner.cc:186] Error getting master registration for 127.0.98.62:36465: OK, instance_id { permanent_uuid: "507fcc3e67694e4ab55eb46004aca4ff" instance_seqno: 1790422342475191 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 11:32:22.977870 13848 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 11:32:22.978205 13924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.966264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47828 (local address 127.0.98.60:34547)
0926 11:32:22.966386 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:22.966393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:22.969307 (+ 2914us) server_negotiation.cc:407] Connection header received
0926 11:32:22.969463 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:22.969466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:22.969513 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:22.969568 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:22.970489 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.971253 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.972399 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.972761 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.972838 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:22.974651 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:22.974655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:22.974657 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:22.974701 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:22.976391 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:22.976867 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:22.976870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:22.976872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:22.976968 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:22.977272 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:22.977275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:22.977277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:22.977459 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:22.977558 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:22.977901 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 11:32:22.978031 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 11:32:22.972934 13774 master_runner.cc:186] Error getting master registration for 127.0.98.62:36465: OK, instance_id { permanent_uuid: "507fcc3e67694e4ab55eb46004aca4ff" instance_seqno: 1790422342475191 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 11:32:22.981124 13774 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 11:32:22.982362 13948 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa065c72738a4c95ac402c16738b1574 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507fcc3e67694e4ab55eb46004aca4ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36465 } } peers { permanent_uuid: "c18bfc13bee2485c8e2e47ee0582e496" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36667 } } peers { permanent_uuid: "aa065c72738a4c95ac402c16738b1574" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34547 } } }
I20260926 11:32:22.982568 13948 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa065c72738a4c95ac402c16738b1574 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:32:22.982473 13767 raft_consensus.cc:348] T 00000000000000000000000000000000 P 507fcc3e67694e4ab55eb46004aca4ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507fcc3e67694e4ab55eb46004aca4ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36465 } } peers { permanent_uuid: "c18bfc13bee2485c8e2e47ee0582e496" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36667 } } peers { permanent_uuid: "aa065c72738a4c95ac402c16738b1574" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34547 } }
I20260926 11:32:22.982713 13767 raft_consensus.cc:374] T 00000000000000000000000000000000 P 507fcc3e67694e4ab55eb46004aca4ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:32:22.982777 13767 raft_consensus.cc:729] T 00000000000000000000000000000000 P 507fcc3e67694e4ab55eb46004aca4ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 507fcc3e67694e4ab55eb46004aca4ff, State: Initialized, Role: FOLLOWER
I20260926 11:32:22.983253 13767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 507fcc3e67694e4ab55eb46004aca4ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507fcc3e67694e4ab55eb46004aca4ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36465 } } peers { permanent_uuid: "c18bfc13bee2485c8e2e47ee0582e496" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36667 } } peers { permanent_uuid: "aa065c72738a4c95ac402c16738b1574" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34547 } }
I20260926 11:32:22.984692 13767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 507fcc3e67694e4ab55eb46004aca4ff [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:32:22.986272 13928 raft_consensus.cc:482] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:32:22.986462 13928 raft_consensus.cc:504] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507fcc3e67694e4ab55eb46004aca4ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36465 } } peers { permanent_uuid: "c18bfc13bee2485c8e2e47ee0582e496" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36667 } } peers { permanent_uuid: "aa065c72738a4c95ac402c16738b1574" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34547 } }
I20260926 11:32:22.987514 13928 leader_election.cc:288] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 507fcc3e67694e4ab55eb46004aca4ff (127.0.98.62:36465), aa065c72738a4c95ac402c16738b1574 (127.0.98.60:34547)
I20260926 11:32:22.988022 13741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c18bfc13bee2485c8e2e47ee0582e496" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "507fcc3e67694e4ab55eb46004aca4ff" is_pre_election: true
I20260926 11:32:22.988130 13893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c18bfc13bee2485c8e2e47ee0582e496" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa065c72738a4c95ac402c16738b1574" is_pre_election: true
I20260926 11:32:22.988322 13741 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 507fcc3e67694e4ab55eb46004aca4ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c18bfc13bee2485c8e2e47ee0582e496 in term 0.
I20260926 11:32:22.988438 13893 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aa065c72738a4c95ac402c16738b1574 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c18bfc13bee2485c8e2e47ee0582e496 in term 0.
I20260926 11:32:22.988759 13793 leader_election.cc:302] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 507fcc3e67694e4ab55eb46004aca4ff, c18bfc13bee2485c8e2e47ee0582e496; no voters:
I20260926 11:32:22.989122 13928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:32:22.989205 13928 raft_consensus.cc:482] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:32:22.989257 13928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:32:22.990378 13928 raft_consensus.cc:504] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507fcc3e67694e4ab55eb46004aca4ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36465 } } peers { permanent_uuid: "c18bfc13bee2485c8e2e47ee0582e496" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36667 } } peers { permanent_uuid: "aa065c72738a4c95ac402c16738b1574" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34547 } }
I20260926 11:32:22.990922 13741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c18bfc13bee2485c8e2e47ee0582e496" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "507fcc3e67694e4ab55eb46004aca4ff"
I20260926 11:32:22.991040 13741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 507fcc3e67694e4ab55eb46004aca4ff [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:32:22.992329 13741 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 507fcc3e67694e4ab55eb46004aca4ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18bfc13bee2485c8e2e47ee0582e496 in term 1.
I20260926 11:32:22.992617 13793 leader_election.cc:302] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 507fcc3e67694e4ab55eb46004aca4ff, c18bfc13bee2485c8e2e47ee0582e496; no voters:
I20260926 11:32:22.992937 13928 leader_election.cc:288] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496 [CANDIDATE]: Term 1 election: Requested vote from peers 507fcc3e67694e4ab55eb46004aca4ff (127.0.98.62:36465), aa065c72738a4c95ac402c16738b1574 (127.0.98.60:34547)
I20260926 11:32:22.993026 13928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:32:22.993091 13928 raft_consensus.cc:686] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496 [term 1 LEADER]: Becoming Leader. State: Replica: c18bfc13bee2485c8e2e47ee0582e496, State: Running, Role: LEADER
I20260926 11:32:22.993326 13893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c18bfc13bee2485c8e2e47ee0582e496" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa065c72738a4c95ac402c16738b1574"
I20260926 11:32:22.993402 13928 consensus_queue.cc:237] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507fcc3e67694e4ab55eb46004aca4ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36465 } } peers { permanent_uuid: "c18bfc13bee2485c8e2e47ee0582e496" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36667 } } peers { permanent_uuid: "aa065c72738a4c95ac402c16738b1574" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34547 } }
I20260926 11:32:22.993467 13893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa065c72738a4c95ac402c16738b1574 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:32:22.994738 13893 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aa065c72738a4c95ac402c16738b1574 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c18bfc13bee2485c8e2e47ee0582e496 in term 1.
I20260926 11:32:22.997611 13699 master_runner.cc:263] No leader master found from master 507fcc3e67694e4ab55eb46004aca4ff
I20260926 11:32:22.997689 13699 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 11:32:22.998231 13974 sys_catalog.cc:455] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c18bfc13bee2485c8e2e47ee0582e496. Latest consensus state: current_term: 1 leader_uuid: "c18bfc13bee2485c8e2e47ee0582e496" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507fcc3e67694e4ab55eb46004aca4ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36465 } } peers { permanent_uuid: "c18bfc13bee2485c8e2e47ee0582e496" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36667 } } peers { permanent_uuid: "aa065c72738a4c95ac402c16738b1574" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34547 } } }
I20260926 11:32:22.998371 13974 sys_catalog.cc:458] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496 [sys.catalog]: This master's current role is: LEADER
I20260926 11:32:22.999294 13771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.965752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41980 (local address 127.0.98.62:36465)
0926 11:32:22.965961 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:22.965965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:22.968466 (+ 2501us) server_negotiation.cc:407] Connection header received
0926 11:32:22.968522 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:22.968524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:22.968565 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:22.968641 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:22.982337 (+ 13696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.983049 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.984177 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.984519 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.984598 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:22.988789 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:22.988793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:22.988796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:22.988823 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:22.996331 (+ 7508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:22.997862 (+ 1531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:22.997865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:22.997867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:22.997969 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:22.998319 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:22.998322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:22.998324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:22.998584 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:22.998694 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:22.999003 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 11:32:22.999122 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260926 11:32:23.002290 13976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:32:23.003409 13976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:32:23.004086 13847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:22.965847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56820 (local address 127.0.98.61:36667)
0926 11:32:22.966040 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:22.966044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:22.966803 (+ 759us) server_negotiation.cc:407] Connection header received
0926 11:32:22.966908 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:22.966913 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:22.966969 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:22.967065 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:22.981423 (+ 14358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.982187 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.996492 (+ 14305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:22.996835 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:22.996914 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:23.000314 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:23.000318 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:23.000320 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:23.000345 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:23.002208 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:23.002717 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:23.002721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:23.002723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:23.002802 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:23.003107 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:23.003111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:23.003113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:23.003321 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:23.003434 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:23.003826 (+ 392us) server_negotiation.cc:299] Negotiation successful
0926 11:32:23.003946 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
W20260926 11:32:23.008989 13973 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 507fcc3e67694e4ab55eb46004aca4ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:32:23.009078 13973 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:32:23.009874 13962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 507fcc3e67694e4ab55eb46004aca4ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507fcc3e67694e4ab55eb46004aca4ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36465 } } peers { permanent_uuid: "c18bfc13bee2485c8e2e47ee0582e496" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36667 } } peers { permanent_uuid: "aa065c72738a4c95ac402c16738b1574" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34547 } } }
I20260926 11:32:23.010025 13962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 507fcc3e67694e4ab55eb46004aca4ff [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:32:23.010803 13741 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 507fcc3e67694e4ab55eb46004aca4ff [term 1 FOLLOWER]: Refusing update from remote peer c18bfc13bee2485c8e2e47ee0582e496: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:32:23.010841 13893 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aa065c72738a4c95ac402c16738b1574 [term 1 FOLLOWER]: Refusing update from remote peer c18bfc13bee2485c8e2e47ee0582e496: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:32:23.011407 13974 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496 [LEADER]: Connected to new peer: Peer: permanent_uuid: "507fcc3e67694e4ab55eb46004aca4ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:32:23.011586 13928 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa065c72738a4c95ac402c16738b1574" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:32:23.015188 13962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 507fcc3e67694e4ab55eb46004aca4ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader c18bfc13bee2485c8e2e47ee0582e496. Latest consensus state: current_term: 1 leader_uuid: "c18bfc13bee2485c8e2e47ee0582e496" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507fcc3e67694e4ab55eb46004aca4ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36465 } } peers { permanent_uuid: "c18bfc13bee2485c8e2e47ee0582e496" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36667 } } peers { permanent_uuid: "aa065c72738a4c95ac402c16738b1574" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34547 } } }
I20260926 11:32:23.015337 13962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 507fcc3e67694e4ab55eb46004aca4ff [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:32:23.015399 13948 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa065c72738a4c95ac402c16738b1574 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c18bfc13bee2485c8e2e47ee0582e496. Latest consensus state: current_term: 1 leader_uuid: "c18bfc13bee2485c8e2e47ee0582e496" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507fcc3e67694e4ab55eb46004aca4ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36465 } } peers { permanent_uuid: "c18bfc13bee2485c8e2e47ee0582e496" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36667 } } peers { permanent_uuid: "aa065c72738a4c95ac402c16738b1574" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34547 } } }
I20260926 11:32:23.015504 13948 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa065c72738a4c95ac402c16738b1574 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:32:23.024928 13977 mvcc.cc:205] Tried to move back new op lower bound from 7333569916963823616 to 7333569916903718912. Current Snapshot: MvccSnapshot[applied={T|T < 7333569916963823616}]
I20260926 11:32:23.025127 13928 sys_catalog.cc:455] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c18bfc13bee2485c8e2e47ee0582e496" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507fcc3e67694e4ab55eb46004aca4ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36465 } } peers { permanent_uuid: "c18bfc13bee2485c8e2e47ee0582e496" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36667 } } peers { permanent_uuid: "aa065c72738a4c95ac402c16738b1574" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34547 } } }
I20260926 11:32:23.025254 13928 sys_catalog.cc:458] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496 [sys.catalog]: This master's current role is: LEADER
I20260926 11:32:23.026504 13962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 507fcc3e67694e4ab55eb46004aca4ff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c18bfc13bee2485c8e2e47ee0582e496" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507fcc3e67694e4ab55eb46004aca4ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36465 } } peers { permanent_uuid: "c18bfc13bee2485c8e2e47ee0582e496" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36667 } } peers { permanent_uuid: "aa065c72738a4c95ac402c16738b1574" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34547 } } }
I20260926 11:32:23.026625 13962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 507fcc3e67694e4ab55eb46004aca4ff [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:32:23.027027 13979 mvcc.cc:205] Tried to move back new op lower bound from 7333569916963823616 to 7333569916903718912. Current Snapshot: MvccSnapshot[applied={T|T < 7333569916963823616}]
I20260926 11:32:23.027299 13974 sys_catalog.cc:455] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c18bfc13bee2485c8e2e47ee0582e496" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507fcc3e67694e4ab55eb46004aca4ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36465 } } peers { permanent_uuid: "c18bfc13bee2485c8e2e47ee0582e496" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36667 } } peers { permanent_uuid: "aa065c72738a4c95ac402c16738b1574" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34547 } } }
I20260926 11:32:23.027422 13974 sys_catalog.cc:458] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496 [sys.catalog]: This master's current role is: LEADER
I20260926 11:32:23.028362 13948 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa065c72738a4c95ac402c16738b1574 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c18bfc13bee2485c8e2e47ee0582e496" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "507fcc3e67694e4ab55eb46004aca4ff" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36465 } } peers { permanent_uuid: "c18bfc13bee2485c8e2e47ee0582e496" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36667 } } peers { permanent_uuid: "aa065c72738a4c95ac402c16738b1574" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34547 } } }
I20260926 11:32:23.028489 13948 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa065c72738a4c95ac402c16738b1574 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:32:23.030241 13980 mvcc.cc:205] Tried to move back new op lower bound from 7333569916963823616 to 7333569916903718912. Current Snapshot: MvccSnapshot[applied={T|T < 7333569916963823616}]
I20260926 11:32:23.030395 13976 catalog_manager.cc:1347] Generated new cluster ID: 9212f053f5a047a5a7398d4eb550d72e
I20260926 11:32:23.030458 13976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:32:23.056514 13976 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:32:23.057838 13976 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:32:23.072770 13976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c18bfc13bee2485c8e2e47ee0582e496: Generated new TSK 0
I20260926 11:32:23.073745 13976 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:32:23.975867 13960 catalog_manager.cc:1259] Loaded cluster ID: 9212f053f5a047a5a7398d4eb550d72e
I20260926 11:32:23.975984 13960 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aa065c72738a4c95ac402c16738b1574: loading cluster ID for follower catalog manager: success
I20260926 11:32:23.977622 13960 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aa065c72738a4c95ac402c16738b1574: acquiring CA information for follower catalog manager: success
I20260926 11:32:23.978845 13960 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aa065c72738a4c95ac402c16738b1574: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 11:32:24.010350 13973 catalog_manager.cc:1259] Loaded cluster ID: 9212f053f5a047a5a7398d4eb550d72e
I20260926 11:32:24.010444 13973 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 507fcc3e67694e4ab55eb46004aca4ff: loading cluster ID for follower catalog manager: success
I20260926 11:32:24.012187 13973 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 507fcc3e67694e4ab55eb46004aca4ff: acquiring CA information for follower catalog manager: success
I20260926 11:32:24.013365 13973 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 507fcc3e67694e4ab55eb46004aca4ff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 11:32:24.144608 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 13699
I20260926 11:32:24.151168 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 13774
I20260926 11:32:24.157141 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 13848
2026-09-26T11:32:24Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3026 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:32:24 dist-test-slave-mrck krb5kdc[13993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:32:24 dist-test-slave-mrck krb5kdc[13993](info): set up 2 sockets
Sep 26 11:32:24 dist-test-slave-mrck krb5kdc[13993](info): commencing operation
krb5kdc: starting...
W20260926 11:32:25.208088 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:32:25 dist-test-slave-mrck krb5kdc[13993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:32:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:32:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:32:25.328250 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36161
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:32:25.436535 14006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:32:25.436800 14006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:32:25.436839 14006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:32:25.440711 14006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:32:25.440763 14006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:32:25.440783 14006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:32:25.440798 14006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:32:25.440812 14006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:32:25.445990 14006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:32:25.447023 14006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:32:25.448001 14006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:32:25.456111 14011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:25.456136 14014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:25.456092 14012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:25.456380 14006 server_base.cc:1034] running on GCE node
I20260926 11:32:25.456946 14006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:32:25.457993 14006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:32:25.459175 14006 hybrid_clock.cc:648] HybridClock initialized: now 1790422345459145 us; error 41 us; skew 500 ppm
Sep 26 11:32:25 dist-test-slave-mrck krb5kdc[13993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:32:25.461906 14006 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:32:25.462728 14006 webserver.cc:460] Webserver started at http://127.0.98.62:34563/ using document root <none> and password file <none>
I20260926 11:32:25.463231 14006 fs_manager.cc:359] Metadata directory not provided
I20260926 11:32:25.463315 14006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:32:25.463546 14006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:32:25.465574 14006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "0d67e4c62f9840178d290d5e52d4a087"
format_stamp: "Formatted at 2026-09-26 11:32:25 on dist-test-slave-mrck"
server_key: "2ea9cc35b370b31b1d3547b15d415d97"
server_key_iv: "086a7376a822885a053a4db2dc7b2e3f"
server_key_version: "encryptionkey@0"
I20260926 11:32:25.466082 14006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "0d67e4c62f9840178d290d5e52d4a087"
format_stamp: "Formatted at 2026-09-26 11:32:25 on dist-test-slave-mrck"
server_key: "2ea9cc35b370b31b1d3547b15d415d97"
server_key_iv: "086a7376a822885a053a4db2dc7b2e3f"
server_key_version: "encryptionkey@0"
I20260926 11:32:25.469745 14006 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 11:32:25.472023 14021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:25.473199 14006 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 11:32:25.473325 14006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "0d67e4c62f9840178d290d5e52d4a087"
format_stamp: "Formatted at 2026-09-26 11:32:25 on dist-test-slave-mrck"
server_key: "2ea9cc35b370b31b1d3547b15d415d97"
server_key_iv: "086a7376a822885a053a4db2dc7b2e3f"
server_key_version: "encryptionkey@0"
I20260926 11:32:25.473428 14006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:32:25.488272 14006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:32:25.491432 14006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:32:25.491639 14006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:32:25.498783 14006 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36161
I20260926 11:32:25.498847 14073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36161 every 8 connection(s)
I20260926 11:32:25.499791 14006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:32:25.502573 14074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:32:25.504195 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 14006
I20260926 11:32:25.504325 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:32:25.504648 392 external_mini_cluster.cc:1428] Setting key 0483e61f995a9931371f6d9b776b77bd
I20260926 11:32:25.509032 14074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d67e4c62f9840178d290d5e52d4a087: Bootstrap starting.
I20260926 11:32:25.511355 14074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d67e4c62f9840178d290d5e52d4a087: Neither blocks nor log segments found. Creating new log.
I20260926 11:32:25.512238 14074 log.cc:824] T 00000000000000000000000000000000 P 0d67e4c62f9840178d290d5e52d4a087: Log is configured to *not* fsync() on all Append() calls
Sep 26 11:32:25 dist-test-slave-mrck krb5kdc[13993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:32:25.514500 14074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d67e4c62f9840178d290d5e52d4a087: No bootstrap required, opened a new log
I20260926 11:32:25.518744 14077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:25.506050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43438 (local address 127.0.98.62:36161)
0926 11:32:25.506479 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:25.506507 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:25.506542 (+ 35us) server_negotiation.cc:407] Connection header received
0926 11:32:25.507170 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:25.507187 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:25.507513 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:25.507872 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:25.508727 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:25.509750 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:25.510703 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:25.511170 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:25.511339 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:25.514264 (+ 2925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:25.514269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:25.514281 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:25.514306 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:25.515974 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:25.516456 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:25.516460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:25.516465 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:25.516576 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:25.516878 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:25.516880 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:25.516881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:25.517235 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:25.517370 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:25.517696 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 11:32:25.518010 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":133,"threads_started":1}
I20260926 11:32:25.518642 14074 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d67e4c62f9840178d290d5e52d4a087 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d67e4c62f9840178d290d5e52d4a087" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36161 } }
I20260926 11:32:25.518868 14074 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d67e4c62f9840178d290d5e52d4a087 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:32:25.518914 14074 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d67e4c62f9840178d290d5e52d4a087 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d67e4c62f9840178d290d5e52d4a087, State: Initialized, Role: FOLLOWER
I20260926 11:32:25.519335 14074 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d67e4c62f9840178d290d5e52d4a087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d67e4c62f9840178d290d5e52d4a087" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36161 } }
I20260926 11:32:25.519459 14074 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d67e4c62f9840178d290d5e52d4a087 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:32:25.519572 14074 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d67e4c62f9840178d290d5e52d4a087 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:32:25.519690 14074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d67e4c62f9840178d290d5e52d4a087 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:32:25.520606 14074 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d67e4c62f9840178d290d5e52d4a087 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d67e4c62f9840178d290d5e52d4a087" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36161 } }
I20260926 11:32:25.520980 14074 leader_election.cc:302] T 00000000000000000000000000000000 P 0d67e4c62f9840178d290d5e52d4a087 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d67e4c62f9840178d290d5e52d4a087; no voters:
I20260926 11:32:25.521309 14074 leader_election.cc:288] T 00000000000000000000000000000000 P 0d67e4c62f9840178d290d5e52d4a087 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:32:25.521415 14079 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d67e4c62f9840178d290d5e52d4a087 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:32:25.521626 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:32:25.521648 14079 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d67e4c62f9840178d290d5e52d4a087 [term 1 LEADER]: Becoming Leader. State: Replica: 0d67e4c62f9840178d290d5e52d4a087, State: Running, Role: LEADER
I20260926 11:32:25.521919 14079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d67e4c62f9840178d290d5e52d4a087 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d67e4c62f9840178d290d5e52d4a087" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36161 } }
I20260926 11:32:25.522615 14074 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d67e4c62f9840178d290d5e52d4a087 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:32:25.523643 14080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d67e4c62f9840178d290d5e52d4a087 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d67e4c62f9840178d290d5e52d4a087" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d67e4c62f9840178d290d5e52d4a087" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36161 } } }
I20260926 11:32:25.523775 14080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d67e4c62f9840178d290d5e52d4a087 [sys.catalog]: This master's current role is: LEADER
I20260926 11:32:25.523970 14081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d67e4c62f9840178d290d5e52d4a087 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d67e4c62f9840178d290d5e52d4a087. Latest consensus state: current_term: 1 leader_uuid: "0d67e4c62f9840178d290d5e52d4a087" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d67e4c62f9840178d290d5e52d4a087" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36161 } } }
I20260926 11:32:25.524067 14081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d67e4c62f9840178d290d5e52d4a087 [sys.catalog]: This master's current role is: LEADER
I20260926 11:32:25.524645 14088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:32:25.527120 14088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:32:25.531970 14088 catalog_manager.cc:1347] Generated new cluster ID: f9b3b8354a3e44a49b5fb1c719440314
I20260926 11:32:25.532032 14088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:32:25.542944 14088 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:32:25.544281 14088 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:32:25.552189 14088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d67e4c62f9840178d290d5e52d4a087: Generated new TSK 0
I20260926 11:32:25.553074 14088 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:32:25.578033 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 14006
2026-09-26T11:32:25Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1418 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:32:25 dist-test-slave-mrck krb5kdc[14101](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:32:25 dist-test-slave-mrck krb5kdc[14101](info): set up 2 sockets
Sep 26 11:32:25 dist-test-slave-mrck krb5kdc[14101](info): commencing operation
krb5kdc: starting...
W20260926 11:32:26.607998 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:32:26 dist-test-slave-mrck krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:32:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:32:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:32:26.727967 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36827
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:32:26.839993 14114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:32:26.840269 14114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:32:26.840309 14114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:32:26.844237 14114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:32:26.844293 14114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:32:26.844316 14114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:32:26.844332 14114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:32:26.844348 14114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:32:26.849700 14114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:32:26.850698 14114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:32:26.851651 14114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:32:26.860134 14119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:26.860133 14120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:26.860303 14122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:26.860414 14114 server_base.cc:1034] running on GCE node
I20260926 11:32:26.861052 14114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:32:26.862226 14114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:32:26.863408 14114 hybrid_clock.cc:648] HybridClock initialized: now 1790422346863397 us; error 35 us; skew 500 ppm
Sep 26 11:32:26 dist-test-slave-mrck krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:32:26.869482 14114 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:32:26.870436 14114 webserver.cc:460] Webserver started at http://127.0.98.62:41119/ using document root <none> and password file <none>
I20260926 11:32:26.871034 14114 fs_manager.cc:359] Metadata directory not provided
I20260926 11:32:26.871129 14114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:32:26.871366 14114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:32:26.873535 14114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "dda02a9ff30c4333b4e9c13a189efbb7"
format_stamp: "Formatted at 2026-09-26 11:32:26 on dist-test-slave-mrck"
server_key: "6c7b1994c5758f624b29c4c34662a7e2"
server_key_iv: "8d3e40550eeb1c57389c2d09306b8673"
server_key_version: "encryptionkey@0"
I20260926 11:32:26.874132 14114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "dda02a9ff30c4333b4e9c13a189efbb7"
format_stamp: "Formatted at 2026-09-26 11:32:26 on dist-test-slave-mrck"
server_key: "6c7b1994c5758f624b29c4c34662a7e2"
server_key_iv: "8d3e40550eeb1c57389c2d09306b8673"
server_key_version: "encryptionkey@0"
I20260926 11:32:26.877882 14114 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 11:32:26.880295 14129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:26.881526 14114 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:32:26.881669 14114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "dda02a9ff30c4333b4e9c13a189efbb7"
format_stamp: "Formatted at 2026-09-26 11:32:26 on dist-test-slave-mrck"
server_key: "6c7b1994c5758f624b29c4c34662a7e2"
server_key_iv: "8d3e40550eeb1c57389c2d09306b8673"
server_key_version: "encryptionkey@0"
I20260926 11:32:26.881783 14114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:32:26.899716 14114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:32:26.903169 14114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:32:26.903409 14114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:32:26.910573 14114 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36827
I20260926 11:32:26.910956 14181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36827 every 8 connection(s)
I20260926 11:32:26.911631 14114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:32:26.914292 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 14114
I20260926 11:32:26.914463 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:32:26.914764 392 external_mini_cluster.cc:1428] Setting key 465133beef5fa5486103eee96c488dc8
I20260926 11:32:26.914765 14182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:32:26.921187 14182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7: Bootstrap starting.
I20260926 11:32:26.923774 14182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7: Neither blocks nor log segments found. Creating new log.
Sep 26 11:32:26 dist-test-slave-mrck krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:32:26.924593 14182 log.cc:824] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7: Log is configured to *not* fsync() on all Append() calls
I20260926 11:32:26.926879 14182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7: No bootstrap required, opened a new log
I20260926 11:32:26.929896 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:26.916222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58798 (local address 127.0.98.62:36827)
0926 11:32:26.916667 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:26.916682 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:26.916716 (+ 34us) server_negotiation.cc:407] Connection header received
0926 11:32:26.917558 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:26.917577 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:26.917931 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:26.918272 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:26.919119 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:26.920177 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:26.921038 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:26.921588 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:26.921761 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:26.924425 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:26.924433 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:26.924453 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:26.924490 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:26.926546 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:26.927029 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:26.927035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:26.927040 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:26.927193 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:26.927467 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:26.927470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:26.927471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:26.928017 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:26.928217 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:26.928495 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 11:32:26.928810 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":140,"threads_started":1}
I20260926 11:32:26.931792 14182 raft_consensus.cc:348] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda02a9ff30c4333b4e9c13a189efbb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36827 } }
I20260926 11:32:26.932050 14182 raft_consensus.cc:374] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:32:26.932122 14182 raft_consensus.cc:729] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dda02a9ff30c4333b4e9c13a189efbb7, State: Initialized, Role: FOLLOWER
I20260926 11:32:26.932580 14182 consensus_queue.cc:260] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda02a9ff30c4333b4e9c13a189efbb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36827 } }
I20260926 11:32:26.932732 14182 raft_consensus.cc:388] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:32:26.932816 14182 raft_consensus.cc:482] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:32:26.932953 14182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:32:26.933907 14182 raft_consensus.cc:504] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda02a9ff30c4333b4e9c13a189efbb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36827 } }
I20260926 11:32:26.934260 14182 leader_election.cc:302] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dda02a9ff30c4333b4e9c13a189efbb7; no voters:
I20260926 11:32:26.934619 14182 leader_election.cc:288] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:32:26.934749 14187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:32:26.934995 14187 raft_consensus.cc:686] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [term 1 LEADER]: Becoming Leader. State: Replica: dda02a9ff30c4333b4e9c13a189efbb7, State: Running, Role: LEADER
I20260926 11:32:26.935295 14187 consensus_queue.cc:237] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda02a9ff30c4333b4e9c13a189efbb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36827 } }
I20260926 11:32:26.935909 14182 sys_catalog.cc:564] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:32:26.936761 14188 sys_catalog.cc:455] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dda02a9ff30c4333b4e9c13a189efbb7. Latest consensus state: current_term: 1 leader_uuid: "dda02a9ff30c4333b4e9c13a189efbb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda02a9ff30c4333b4e9c13a189efbb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36827 } } }
I20260926 11:32:26.936949 14188 sys_catalog.cc:458] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [sys.catalog]: This master's current role is: LEADER
I20260926 11:32:26.937023 14189 sys_catalog.cc:455] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dda02a9ff30c4333b4e9c13a189efbb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda02a9ff30c4333b4e9c13a189efbb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36827 } } }
I20260926 11:32:26.937258 14189 sys_catalog.cc:458] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [sys.catalog]: This master's current role is: LEADER
I20260926 11:32:26.937592 14196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:32:26.940688 14196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:32:26.946583 14196 catalog_manager.cc:1347] Generated new cluster ID: 2b2bd1c2a3be49eb92a27f7accacafe6
I20260926 11:32:26.946666 14196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:32:26.963057 14196 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:32:26.964382 14196 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:32:26.970867 14196 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7: Generated new TSK 0
I20260926 11:32:26.971678 14196 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:32:27.006908 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36827
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:32:27.120528 14208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:32:27.120810 14208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:32:27.120853 14208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:32:27.124927 14208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:32:27.124990 14208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:32:27.125077 14208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:32:27.130460 14208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:32:27.131539 14208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:32:27.132635 14208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 11:32:27.142266 14208 server_base.cc:1034] running on GCE node
W20260926 11:32:27.142210 14214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:27.142235 14213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:27.142215 14216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:27.142973 14208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:32:27.143594 14208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:32:27.144747 14208 hybrid_clock.cc:648] HybridClock initialized: now 1790422347144729 us; error 33 us; skew 500 ppm
Sep 26 11:32:27 dist-test-slave-mrck krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:32:27.147861 14208 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:32:27.148738 14208 webserver.cc:460] Webserver started at http://127.0.98.1:44801/ using document root <none> and password file <none>
I20260926 11:32:27.149319 14208 fs_manager.cc:359] Metadata directory not provided
I20260926 11:32:27.149379 14208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:32:27.149610 14208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:32:27.151657 14208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/data/instance:
uuid: "b611f7b8e9584eb4928bd774e1721703"
format_stamp: "Formatted at 2026-09-26 11:32:27 on dist-test-slave-mrck"
server_key: "ab53f151e16d9bb0cf2a775b4cb576c8"
server_key_iv: "46b608683cee621e331c5681dfed117d"
server_key_version: "encryptionkey@0"
I20260926 11:32:27.152304 14208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b611f7b8e9584eb4928bd774e1721703"
format_stamp: "Formatted at 2026-09-26 11:32:27 on dist-test-slave-mrck"
server_key: "ab53f151e16d9bb0cf2a775b4cb576c8"
server_key_iv: "46b608683cee621e331c5681dfed117d"
server_key_version: "encryptionkey@0"
I20260926 11:32:27.156237 14208 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:32:27.158703 14223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:27.160012 14208 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:32:27.160136 14208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "b611f7b8e9584eb4928bd774e1721703"
format_stamp: "Formatted at 2026-09-26 11:32:27 on dist-test-slave-mrck"
server_key: "ab53f151e16d9bb0cf2a775b4cb576c8"
server_key_iv: "46b608683cee621e331c5681dfed117d"
server_key_version: "encryptionkey@0"
I20260926 11:32:27.160243 14208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:32:27.175117 14208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:32:27.178171 14208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:32:27.178380 14208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:32:27.178966 14208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:32:27.179867 14208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:32:27.179934 14208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:27.180011 14208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:32:27.180053 14208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:27.189390 14208 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39175
I20260926 11:32:27.189412 14336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39175 every 8 connection(s)
I20260926 11:32:27.190299 14208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:32:27.193663 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 14208
I20260926 11:32:27.193806 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:32:27.194087 392 external_mini_cluster.cc:1428] Setting key 8179db7bcb47b19ae5005d71669f5ce2
Sep 26 11:32:27 dist-test-slave-mrck krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 11:32:27.206094 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:27.192203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54479 (local address 127.0.98.62:36827)
0926 11:32:27.192337 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:27.192341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:27.193155 (+ 814us) server_negotiation.cc:407] Connection header received
0926 11:32:27.194641 (+ 1486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:27.194644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:27.194696 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:27.194795 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:27.196273 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:27.197153 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:27.198934 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:27.199166 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:27.199237 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:27.202125 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:27.202129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:27.202132 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:27.202162 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:27.203931 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:27.204563 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:27.204567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:27.204569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:27.204661 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:27.205023 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:27.205026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:27.205027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:27.205266 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:27.205346 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:27.205851 (+ 505us) server_negotiation.cc:299] Negotiation successful
0926 11:32:27.205958 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 11:32:27.207058 14339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:27.192591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.1:54479)
0926 11:32:27.193010 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:27.193052 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:27.194366 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:27.194977 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:27.194986 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:27.195560 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:32:27.196128 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:27.196142 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:27.197298 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:27.197304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:27.198806 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:27.198820 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:27.199314 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:27.199320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:27.199556 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:27.200215 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:32:27.200237 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:32:27.201938 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:32:27.204083 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:27.204087 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:27.204100 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:27.204458 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:27.204750 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:27.204752 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:27.204754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:27.204908 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:27.205388 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:32:27.205393 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:32:27.205641 (+ 248us) client_negotiation.cc:769] Sending connection context
0926 11:32:27.205805 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 11:32:27.206074 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":115,"threads_started":1}
I20260926 11:32:27.208623 14337 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36827
I20260926 11:32:27.208920 14337 heartbeater.cc:464] Registering TS with master...
I20260926 11:32:27.209517 14337 heartbeater.cc:511] Master 127.0.98.62:36827 requested a full tablet report, sending...
I20260926 11:32:27.210996 14146 ts_manager.cc:194] Registered new tserver with Master: b611f7b8e9584eb4928bd774e1721703 (127.0.98.1:39175)
I20260926 11:32:27.212181 14146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54479
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:32:27.222410 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36827
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:32:27.333096 14342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:32:27.333379 14342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:32:27.333431 14342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:32:27.338023 14342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:32:27.338090 14342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:32:27.338179 14342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:32:27.343581 14342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:32:27.344718 14342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:32:27.345779 14342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:32:27.355557 14348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:27.355562 14347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:27.355610 14342 server_base.cc:1034] running on GCE node
W20260926 11:32:27.355561 14350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:27.356340 14342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:32:27.356964 14342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:32:27.358129 14342 hybrid_clock.cc:648] HybridClock initialized: now 1790422347358107 us; error 33 us; skew 500 ppm
Sep 26 11:32:27 dist-test-slave-mrck krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:32:27.360841 14342 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:32:27.361757 14342 webserver.cc:460] Webserver started at http://127.0.98.2:43129/ using document root <none> and password file <none>
I20260926 11:32:27.362313 14342 fs_manager.cc:359] Metadata directory not provided
I20260926 11:32:27.362371 14342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:32:27.362592 14342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:32:27.364758 14342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/data/instance:
uuid: "53b1888e4c9546a5af572aa5c87b29e2"
format_stamp: "Formatted at 2026-09-26 11:32:27 on dist-test-slave-mrck"
server_key: "a99e397585403845e9ed6288843965f3"
server_key_iv: "51e708aff71c1d1fac7ec47c3e127ce9"
server_key_version: "encryptionkey@0"
I20260926 11:32:27.365317 14342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "53b1888e4c9546a5af572aa5c87b29e2"
format_stamp: "Formatted at 2026-09-26 11:32:27 on dist-test-slave-mrck"
server_key: "a99e397585403845e9ed6288843965f3"
server_key_iv: "51e708aff71c1d1fac7ec47c3e127ce9"
server_key_version: "encryptionkey@0"
I20260926 11:32:27.369092 14342 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:32:27.371552 14357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:27.372892 14342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:32:27.373029 14342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "53b1888e4c9546a5af572aa5c87b29e2"
format_stamp: "Formatted at 2026-09-26 11:32:27 on dist-test-slave-mrck"
server_key: "a99e397585403845e9ed6288843965f3"
server_key_iv: "51e708aff71c1d1fac7ec47c3e127ce9"
server_key_version: "encryptionkey@0"
I20260926 11:32:27.373142 14342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:32:27.386907 14342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:32:27.390101 14342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:32:27.390308 14342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:32:27.390935 14342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:32:27.391867 14342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:32:27.391947 14342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:27.392020 14342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:32:27.392055 14342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:27.400635 14342 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36243
I20260926 11:32:27.400705 14470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36243 every 8 connection(s)
I20260926 11:32:27.401551 14342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:32:27.408754 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 14342
I20260926 11:32:27.408903 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:32:27.409192 392 external_mini_cluster.cc:1428] Setting key 83b4135faf6a126fc3c748a2ae134fd9
Sep 26 11:32:27 dist-test-slave-mrck krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:32:27.416290 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:27.403398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54337 (local address 127.0.98.62:36827)
0926 11:32:27.403576 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:27.403580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:27.404411 (+ 831us) server_negotiation.cc:407] Connection header received
0926 11:32:27.405698 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:27.405701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:27.405755 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:27.405849 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:27.406997 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:27.407538 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:27.408749 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:27.409092 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:27.409208 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:27.412193 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:27.412197 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:27.412199 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:27.412231 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:27.413838 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:27.414410 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:27.414417 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:27.414422 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:27.414529 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:27.414921 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:27.414927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:27.414932 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:27.415204 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:27.415312 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:27.415883 (+ 571us) server_negotiation.cc:299] Negotiation successful
0926 11:32:27.416045 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 11:32:27.417003 14473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:27.403822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.2:54337)
0926 11:32:27.404259 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:27.404301 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:27.405489 (+ 1188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:27.406007 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:27.406014 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:27.406370 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:32:27.406867 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:27.406880 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:27.407645 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:27.407647 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:27.408614 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:27.408623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:27.409247 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:27.409251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:27.409427 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:27.410128 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:32:27.410147 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:32:27.412030 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:32:27.413999 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:27.414006 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:27.414023 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:27.414291 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:27.414629 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:27.414631 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:27.414633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:27.414770 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:27.415315 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:32:27.415321 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:32:27.415592 (+ 271us) client_negotiation.cc:769] Sending connection context
0926 11:32:27.415809 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 11:32:27.416093 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":115,"threads_started":1}
I20260926 11:32:27.418280 14471 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36827
I20260926 11:32:27.418548 14471 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 11:32:27.419119 14471 heartbeater.cc:511] Master 127.0.98.62:36827 requested a full tablet report, sending...
I20260926 11:32:27.420284 14146 ts_manager.cc:194] Registered new tserver with Master: 53b1888e4c9546a5af572aa5c87b29e2 (127.0.98.2:36243)
I20260926 11:32:27.420837 14146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54337
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:32:27.437212 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36827
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:32:27.548462 14476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:32:27.548738 14476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:32:27.548784 14476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:32:27.552906 14476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:32:27.552982 14476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:32:27.553077 14476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:32:27.558444 14476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:32:27.559507 14476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:32:27.560566 14476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:32:27.570199 14482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:27.570225 14484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:27.570204 14481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:27.570614 14476 server_base.cc:1034] running on GCE node
I20260926 11:32:27.571069 14476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:32:27.571820 14476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:32:27.573035 14476 hybrid_clock.cc:648] HybridClock initialized: now 1790422347573004 us; error 56 us; skew 500 ppm
Sep 26 11:32:27 dist-test-slave-mrck krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:32:27.575986 14476 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:32:27.576869 14476 webserver.cc:460] Webserver started at http://127.0.98.3:38563/ using document root <none> and password file <none>
I20260926 11:32:27.577493 14476 fs_manager.cc:359] Metadata directory not provided
I20260926 11:32:27.577582 14476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:32:27.577813 14476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:32:27.579890 14476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/data/instance:
uuid: "133399fee1e447aba4e74af5695a6dd5"
format_stamp: "Formatted at 2026-09-26 11:32:27 on dist-test-slave-mrck"
server_key: "2faa3156c063d6d21e6fc434c83d08ed"
server_key_iv: "46589ce97cb51263d1ecd1b03224a41b"
server_key_version: "encryptionkey@0"
I20260926 11:32:27.580449 14476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "133399fee1e447aba4e74af5695a6dd5"
format_stamp: "Formatted at 2026-09-26 11:32:27 on dist-test-slave-mrck"
server_key: "2faa3156c063d6d21e6fc434c83d08ed"
server_key_iv: "46589ce97cb51263d1ecd1b03224a41b"
server_key_version: "encryptionkey@0"
I20260926 11:32:27.584300 14476 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 11:32:27.586731 14491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:27.588074 14476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:32:27.588207 14476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "133399fee1e447aba4e74af5695a6dd5"
format_stamp: "Formatted at 2026-09-26 11:32:27 on dist-test-slave-mrck"
server_key: "2faa3156c063d6d21e6fc434c83d08ed"
server_key_iv: "46589ce97cb51263d1ecd1b03224a41b"
server_key_version: "encryptionkey@0"
I20260926 11:32:27.588331 14476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:32:27.611536 14476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:32:27.615141 14476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:32:27.615350 14476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:32:27.616043 14476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:32:27.616902 14476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:32:27.616974 14476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:27.617046 14476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:32:27.617090 14476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:27.626528 14476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37853
I20260926 11:32:27.626565 14604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37853 every 8 connection(s)
I20260926 11:32:27.627463 14476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:32:27.633657 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 14476
I20260926 11:32:27.633822 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:32:27.634114 392 external_mini_cluster.cc:1428] Setting key 05801b7cea49fcf83445ee1ee21722c7
Sep 26 11:32:27 dist-test-slave-mrck krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:32:27.641861 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:27.629353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43823 (local address 127.0.98.62:36827)
0926 11:32:27.629500 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:27.629505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:27.630394 (+ 889us) server_negotiation.cc:407] Connection header received
0926 11:32:27.631691 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:27.631694 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:27.631798 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:27.631902 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:27.633127 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:27.633688 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:27.635209 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:27.635495 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:27.635591 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:27.638390 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:27.638393 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:27.638396 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:27.638428 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:27.639787 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:27.640304 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:27.640308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:27.640310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:27.640400 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:27.640736 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:27.640741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:27.640742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:27.640939 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:27.641043 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:27.641553 (+ 510us) server_negotiation.cc:299] Negotiation successful
0926 11:32:27.641679 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 11:32:27.642685 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:27.629782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.3:43823)
0926 11:32:27.630223 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:27.630267 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:27.631459 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:27.632116 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:27.632124 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:27.632497 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:32:27.632973 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:27.632998 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:27.633829 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:27.633833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:27.635077 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:27.635088 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:27.635639 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:27.635643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:27.635877 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:27.636620 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:32:27.636637 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:32:27.638212 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:32:27.639906 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:27.639914 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:27.639934 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:27.640197 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:27.640499 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:27.640502 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:27.640504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:27.640640 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:27.641050 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:32:27.641059 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:32:27.641306 (+ 247us) client_negotiation.cc:769] Sending connection context
0926 11:32:27.641502 (+ 196us) client_negotiation.cc:240] Negotiation successful
0926 11:32:27.641867 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":120,"threads_started":1}
I20260926 11:32:27.643817 14605 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36827
I20260926 11:32:27.644068 14605 heartbeater.cc:464] Registering TS with master...
I20260926 11:32:27.644562 14605 heartbeater.cc:511] Master 127.0.98.62:36827 requested a full tablet report, sending...
I20260926 11:32:27.645733 14146 ts_manager.cc:194] Registered new tserver with Master: 133399fee1e447aba4e74af5695a6dd5 (127.0.98.3:37853)
I20260926 11:32:27.646335 14146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43823
I20260926 11:32:27.648195 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:32:27.657327 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:27.650078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58800 (local address 127.0.98.62:36827)
0926 11:32:27.650206 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:27.650211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:27.650387 (+ 176us) server_negotiation.cc:407] Connection header received
0926 11:32:27.650560 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:27.650562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:27.650600 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:27.650680 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:27.651379 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:27.651960 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:27.652797 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:27.653024 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:27.653138 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:27.654471 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:27.654475 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:27.654478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:27.654509 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:27.655633 (+ 1124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:27.656087 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:27.656092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:27.656094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:27.656174 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:27.656446 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:27.656449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:27.656451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:27.656670 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:27.656809 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:27.657045 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 11:32:27.657159 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 11:32:27.660017 14146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:32:27.663046 14146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:32:27.677837 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:27.671752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36243 (local address 127.0.0.1:37640)
0926 11:32:27.672676 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:27.672726 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:27.672854 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:27.673444 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:27.673449 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:27.673471 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:32:27.673648 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:27.673656 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:27.675139 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:27.675142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:27.676370 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:27.676376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:27.677240 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:27.677244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:27.677358 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:27.677393 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 11:32:27.677499 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 11:32:27.677649 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":21,"thread_start_us":92,"threads_started":1}
I20260926 11:32:27.677947 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:27.672000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34012 (local address 127.0.98.1:39175)
0926 11:32:27.672441 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:27.672458 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:27.673259 (+ 801us) server_negotiation.cc:407] Connection header received
0926 11:32:27.673324 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:27.673328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:27.673527 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:27.673680 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:32:27.673989 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:27.675083 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:27.676585 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:27.677118 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:27.677243 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:27.677394 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:27.677692 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 11:32:27.677799 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":70,"threads_started":1}
I20260926 11:32:27.677837 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:27.671756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39175 (local address 127.0.0.1:34012)
0926 11:32:27.673076 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:27.673088 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:27.673166 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:27.673657 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:27.673660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:27.673684 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:32:27.673859 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:27.673866 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:27.675208 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:27.675212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:27.676462 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:27.676472 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:27.677450 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:27.677457 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:27.677552 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:27.677572 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 11:32:27.677642 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 11:32:27.677693 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1262,"mutex_wait_us":466,"spinlock_wait_cycles":7680,"thread_start_us":95,"threads_started":1}
I20260926 11:32:27.678320 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:27.671942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37640 (local address 127.0.98.2:36243)
0926 11:32:27.672651 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:27.672665 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:27.672999 (+ 334us) server_negotiation.cc:407] Connection header received
0926 11:32:27.673070 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:27.673075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:27.673282 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:27.673446 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:32:27.673781 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:27.675007 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:27.676476 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:27.677080 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:27.677229 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:27.677393 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:27.678066 (+ 673us) server_negotiation.cc:299] Negotiation successful
0926 11:32:27.678161 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":167,"threads_started":1}
I20260926 11:32:27.679235 14619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:27.672075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37853 (local address 127.0.0.1:49512)
0926 11:32:27.672785 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:27.672800 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:27.672900 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:27.673650 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:27.673653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:27.673679 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:32:27.673862 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:27.673867 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:27.674880 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:27.674887 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:27.676359 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:27.676370 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:27.678604 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:27.678608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:27.678705 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:27.678731 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 11:32:27.679018 (+ 287us) client_negotiation.cc:240] Negotiation successful
0926 11:32:27.679089 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":116,"spinlock_wait_cycles":384,"thread_start_us":80,"threads_started":1}
I20260926 11:32:27.679762 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:27.672568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49512 (local address 127.0.98.3:37853)
0926 11:32:27.672943 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:27.672955 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:27.672979 (+ 24us) server_negotiation.cc:407] Connection header received
0926 11:32:27.673070 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:27.673075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:27.673282 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:27.673446 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:32:27.674025 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:27.674753 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:27.677479 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:27.678286 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:27.678391 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:27.678537 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:27.679508 (+ 971us) server_negotiation.cc:299] Negotiation successful
0926 11:32:27.679606 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":94,"threads_started":1}
I20260926 11:32:27.681195 14271 tablet_service.cc:1467] Processing CreateTablet for tablet 8248317c305d4f13b8c66cd548bb3072 (DEFAULT_TABLE table=test-table [id=d72c2908348c4a2aa6c0ebb4460bc603]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:32:27.681231 14405 tablet_service.cc:1467] Processing CreateTablet for tablet 8248317c305d4f13b8c66cd548bb3072 (DEFAULT_TABLE table=test-table [id=d72c2908348c4a2aa6c0ebb4460bc603]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:32:27.682163 14271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8248317c305d4f13b8c66cd548bb3072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:32:27.682202 14405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8248317c305d4f13b8c66cd548bb3072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:32:27.682130 14539 tablet_service.cc:1467] Processing CreateTablet for tablet 8248317c305d4f13b8c66cd548bb3072 (DEFAULT_TABLE table=test-table [id=d72c2908348c4a2aa6c0ebb4460bc603]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:32:27.683066 14539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8248317c305d4f13b8c66cd548bb3072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:32:27.688719 14622 tablet_bootstrap.cc:492] T 8248317c305d4f13b8c66cd548bb3072 P b611f7b8e9584eb4928bd774e1721703: Bootstrap starting.
I20260926 11:32:27.690191 14623 tablet_bootstrap.cc:492] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2: Bootstrap starting.
I20260926 11:32:27.691424 14622 tablet_bootstrap.cc:654] T 8248317c305d4f13b8c66cd548bb3072 P b611f7b8e9584eb4928bd774e1721703: Neither blocks nor log segments found. Creating new log.
I20260926 11:32:27.692836 14623 tablet_bootstrap.cc:654] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2: Neither blocks nor log segments found. Creating new log.
I20260926 11:32:27.692839 14622 log.cc:824] T 8248317c305d4f13b8c66cd548bb3072 P b611f7b8e9584eb4928bd774e1721703: Log is configured to *not* fsync() on all Append() calls
I20260926 11:32:27.693313 14624 tablet_bootstrap.cc:492] T 8248317c305d4f13b8c66cd548bb3072 P 133399fee1e447aba4e74af5695a6dd5: Bootstrap starting.
I20260926 11:32:27.695169 14622 tablet_bootstrap.cc:492] T 8248317c305d4f13b8c66cd548bb3072 P b611f7b8e9584eb4928bd774e1721703: No bootstrap required, opened a new log
I20260926 11:32:27.695384 14622 ts_tablet_manager.cc:1397] T 8248317c305d4f13b8c66cd548bb3072 P b611f7b8e9584eb4928bd774e1721703: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20260926 11:32:27.695475 14624 tablet_bootstrap.cc:654] T 8248317c305d4f13b8c66cd548bb3072 P 133399fee1e447aba4e74af5695a6dd5: Neither blocks nor log segments found. Creating new log.
I20260926 11:32:27.696276 14623 log.cc:824] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2: Log is configured to *not* fsync() on all Append() calls
I20260926 11:32:27.696682 14624 log.cc:824] T 8248317c305d4f13b8c66cd548bb3072 P 133399fee1e447aba4e74af5695a6dd5: Log is configured to *not* fsync() on all Append() calls
I20260926 11:32:27.698233 14623 tablet_bootstrap.cc:492] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2: No bootstrap required, opened a new log
I20260926 11:32:27.698457 14623 ts_tablet_manager.cc:1397] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260926 11:32:27.698588 14624 tablet_bootstrap.cc:492] T 8248317c305d4f13b8c66cd548bb3072 P 133399fee1e447aba4e74af5695a6dd5: No bootstrap required, opened a new log
I20260926 11:32:27.698771 14624 ts_tablet_manager.cc:1397] T 8248317c305d4f13b8c66cd548bb3072 P 133399fee1e447aba4e74af5695a6dd5: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260926 11:32:27.699501 14622 raft_consensus.cc:348] T 8248317c305d4f13b8c66cd548bb3072 P b611f7b8e9584eb4928bd774e1721703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b1888e4c9546a5af572aa5c87b29e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36243 } } peers { permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 } } peers { permanent_uuid: "133399fee1e447aba4e74af5695a6dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37853 } }
I20260926 11:32:27.699796 14622 raft_consensus.cc:374] T 8248317c305d4f13b8c66cd548bb3072 P b611f7b8e9584eb4928bd774e1721703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:32:27.699846 14622 raft_consensus.cc:729] T 8248317c305d4f13b8c66cd548bb3072 P b611f7b8e9584eb4928bd774e1721703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b611f7b8e9584eb4928bd774e1721703, State: Initialized, Role: FOLLOWER
I20260926 11:32:27.700342 14622 consensus_queue.cc:260] T 8248317c305d4f13b8c66cd548bb3072 P b611f7b8e9584eb4928bd774e1721703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b1888e4c9546a5af572aa5c87b29e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36243 } } peers { permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 } } peers { permanent_uuid: "133399fee1e447aba4e74af5695a6dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37853 } }
I20260926 11:32:27.701371 14622 ts_tablet_manager.cc:1428] T 8248317c305d4f13b8c66cd548bb3072 P b611f7b8e9584eb4928bd774e1721703: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260926 11:32:27.701762 14337 heartbeater.cc:503] Master 127.0.98.62:36827 was elected leader, sending a full tablet report...
I20260926 11:32:27.702013 14623 raft_consensus.cc:348] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b1888e4c9546a5af572aa5c87b29e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36243 } } peers { permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 } } peers { permanent_uuid: "133399fee1e447aba4e74af5695a6dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37853 } }
I20260926 11:32:27.702260 14623 raft_consensus.cc:374] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:32:27.702306 14623 raft_consensus.cc:729] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53b1888e4c9546a5af572aa5c87b29e2, State: Initialized, Role: FOLLOWER
I20260926 11:32:27.702319 14624 raft_consensus.cc:348] T 8248317c305d4f13b8c66cd548bb3072 P 133399fee1e447aba4e74af5695a6dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b1888e4c9546a5af572aa5c87b29e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36243 } } peers { permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 } } peers { permanent_uuid: "133399fee1e447aba4e74af5695a6dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37853 } }
I20260926 11:32:27.702543 14624 raft_consensus.cc:374] T 8248317c305d4f13b8c66cd548bb3072 P 133399fee1e447aba4e74af5695a6dd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:32:27.702593 14624 raft_consensus.cc:729] T 8248317c305d4f13b8c66cd548bb3072 P 133399fee1e447aba4e74af5695a6dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133399fee1e447aba4e74af5695a6dd5, State: Initialized, Role: FOLLOWER
I20260926 11:32:27.702740 14623 consensus_queue.cc:260] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b1888e4c9546a5af572aa5c87b29e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36243 } } peers { permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 } } peers { permanent_uuid: "133399fee1e447aba4e74af5695a6dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37853 } }
I20260926 11:32:27.703020 14624 consensus_queue.cc:260] T 8248317c305d4f13b8c66cd548bb3072 P 133399fee1e447aba4e74af5695a6dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b1888e4c9546a5af572aa5c87b29e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36243 } } peers { permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 } } peers { permanent_uuid: "133399fee1e447aba4e74af5695a6dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37853 } }
I20260926 11:32:27.703625 14605 heartbeater.cc:503] Master 127.0.98.62:36827 was elected leader, sending a full tablet report...
I20260926 11:32:27.703678 14623 ts_tablet_manager.cc:1428] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 11:32:27.703965 14624 ts_tablet_manager.cc:1428] T 8248317c305d4f13b8c66cd548bb3072 P 133399fee1e447aba4e74af5695a6dd5: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260926 11:32:27.704021 14471 heartbeater.cc:503] Master 127.0.98.62:36827 was elected leader, sending a full tablet report...
W20260926 11:32:27.878532 14606 tablet.cc:2369] T 8248317c305d4f13b8c66cd548bb3072 P 133399fee1e447aba4e74af5695a6dd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 11:32:27.902731 14472 tablet.cc:2369] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 11:32:27.941854 14338 tablet.cc:2369] T 8248317c305d4f13b8c66cd548bb3072 P b611f7b8e9584eb4928bd774e1721703: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:32:28.060794 14629 raft_consensus.cc:482] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:32:28.061000 14629 raft_consensus.cc:504] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b1888e4c9546a5af572aa5c87b29e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36243 } } peers { permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 } } peers { permanent_uuid: "133399fee1e447aba4e74af5695a6dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37853 } }
I20260926 11:32:28.062193 14629 leader_election.cc:288] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b611f7b8e9584eb4928bd774e1721703 (127.0.98.1:39175), 133399fee1e447aba4e74af5695a6dd5 (127.0.98.3:37853)
I20260926 11:32:28.067749 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:28.062537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38621 (local address 127.0.98.1:39175)
0926 11:32:28.062674 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:28.062679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:28.062864 (+ 185us) server_negotiation.cc:407] Connection header received
0926 11:32:28.063024 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:28.063027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:28.063082 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:28.063174 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:32:28.063635 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:28.064448 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:28.066372 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:28.066987 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:28.067111 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:28.067177 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:28.067513 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 11:32:28.067592 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 11:32:28.067750 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:28.062625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39971 (local address 127.0.98.3:37853)
0926 11:32:28.062757 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:28.062761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:28.062938 (+ 177us) server_negotiation.cc:407] Connection header received
0926 11:32:28.063023 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:28.063026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:28.063083 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:28.063174 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:32:28.063635 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:28.064486 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:28.066372 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:28.066990 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:28.067111 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:28.067177 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:28.067528 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 11:32:28.067595 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 11:32:28.067823 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:28.062403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39175 (local address 127.0.98.2:38621)
0926 11:32:28.062739 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:28.062755 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:28.062902 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:28.063237 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:28.063240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:28.063284 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:32:28.063508 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:28.063515 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:28.064620 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:28.064624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:28.066230 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:28.066244 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:28.067133 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:28.067137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:28.067323 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:28.067399 (+ 76us) client_negotiation.cc:769] Sending connection context
0926 11:32:28.067481 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 11:32:28.067615 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":316,"spinlock_wait_cycles":28928,"thread_start_us":87,"threads_started":1}
I20260926 11:32:28.067826 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:28.062495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37853 (local address 127.0.98.2:39971)
0926 11:32:28.062838 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:28.062850 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:28.062925 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:28.063237 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:28.063240 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:28.063285 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:32:28.063508 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:28.063515 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:28.064620 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:28.064624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:28.066242 (+ 1618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:28.066248 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:28.067133 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:28.067137 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:28.067364 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:28.067405 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 11:32:28.067481 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 11:32:28.067620 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":324,"spinlock_wait_cycles":10624,"thread_start_us":82,"threads_started":1}
I20260926 11:32:28.068652 14291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8248317c305d4f13b8c66cd548bb3072" candidate_uuid: "53b1888e4c9546a5af572aa5c87b29e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b611f7b8e9584eb4928bd774e1721703" is_pre_election: true
I20260926 11:32:28.068648 14559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8248317c305d4f13b8c66cd548bb3072" candidate_uuid: "53b1888e4c9546a5af572aa5c87b29e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "133399fee1e447aba4e74af5695a6dd5" is_pre_election: true
I20260926 11:32:28.068972 14559 raft_consensus.cc:2454] T 8248317c305d4f13b8c66cd548bb3072 P 133399fee1e447aba4e74af5695a6dd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53b1888e4c9546a5af572aa5c87b29e2 in term 0.
I20260926 11:32:28.068972 14291 raft_consensus.cc:2454] T 8248317c305d4f13b8c66cd548bb3072 P b611f7b8e9584eb4928bd774e1721703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53b1888e4c9546a5af572aa5c87b29e2 in term 0.
I20260926 11:32:28.069520 14360 leader_election.cc:302] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 133399fee1e447aba4e74af5695a6dd5, 53b1888e4c9546a5af572aa5c87b29e2, b611f7b8e9584eb4928bd774e1721703; no voters:
I20260926 11:32:28.069833 14629 raft_consensus.cc:2781] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:32:28.069942 14629 raft_consensus.cc:482] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:32:28.070026 14629 raft_consensus.cc:3037] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:32:28.071079 14629 raft_consensus.cc:504] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b1888e4c9546a5af572aa5c87b29e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36243 } } peers { permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 } } peers { permanent_uuid: "133399fee1e447aba4e74af5695a6dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37853 } }
I20260926 11:32:28.071470 14629 leader_election.cc:288] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2 [CANDIDATE]: Term 1 election: Requested vote from peers b611f7b8e9584eb4928bd774e1721703 (127.0.98.1:39175), 133399fee1e447aba4e74af5695a6dd5 (127.0.98.3:37853)
I20260926 11:32:28.071884 14291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8248317c305d4f13b8c66cd548bb3072" candidate_uuid: "53b1888e4c9546a5af572aa5c87b29e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b611f7b8e9584eb4928bd774e1721703"
I20260926 11:32:28.071946 14559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8248317c305d4f13b8c66cd548bb3072" candidate_uuid: "53b1888e4c9546a5af572aa5c87b29e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "133399fee1e447aba4e74af5695a6dd5"
I20260926 11:32:28.072022 14291 raft_consensus.cc:3037] T 8248317c305d4f13b8c66cd548bb3072 P b611f7b8e9584eb4928bd774e1721703 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:32:28.072119 14559 raft_consensus.cc:3037] T 8248317c305d4f13b8c66cd548bb3072 P 133399fee1e447aba4e74af5695a6dd5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:32:28.073100 14291 raft_consensus.cc:2454] T 8248317c305d4f13b8c66cd548bb3072 P b611f7b8e9584eb4928bd774e1721703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53b1888e4c9546a5af572aa5c87b29e2 in term 1.
I20260926 11:32:28.073148 14559 raft_consensus.cc:2454] T 8248317c305d4f13b8c66cd548bb3072 P 133399fee1e447aba4e74af5695a6dd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53b1888e4c9546a5af572aa5c87b29e2 in term 1.
I20260926 11:32:28.073396 14360 leader_election.cc:302] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53b1888e4c9546a5af572aa5c87b29e2, b611f7b8e9584eb4928bd774e1721703; no voters:
I20260926 11:32:28.073570 14629 raft_consensus.cc:2781] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:32:28.073803 14629 raft_consensus.cc:686] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2 [term 1 LEADER]: Becoming Leader. State: Replica: 53b1888e4c9546a5af572aa5c87b29e2, State: Running, Role: LEADER
I20260926 11:32:28.074149 14629 consensus_queue.cc:237] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b1888e4c9546a5af572aa5c87b29e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36243 } } peers { permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 } } peers { permanent_uuid: "133399fee1e447aba4e74af5695a6dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37853 } }
I20260926 11:32:28.077551 14146 catalog_manager.cc:5495] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 53b1888e4c9546a5af572aa5c87b29e2 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "53b1888e4c9546a5af572aa5c87b29e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b1888e4c9546a5af572aa5c87b29e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36243 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "133399fee1e447aba4e74af5695a6dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37853 } health_report { overall_health: UNKNOWN } } }
I20260926 11:32:28.129546 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:28.125953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37652 (local address 127.0.98.2:36243)
0926 11:32:28.126101 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:28.126105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:28.126122 (+ 17us) server_negotiation.cc:407] Connection header received
0926 11:32:28.126223 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:28.126226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:28.126283 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:28.126401 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:32:28.126642 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:28.127164 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:28.128043 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:28.128268 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:28.128380 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:28.128687 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:32:28.129138 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:32:28.129275 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:28.129316 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 11:32:28.129380 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 11:32:28.135495 14559 raft_consensus.cc:1261] T 8248317c305d4f13b8c66cd548bb3072 P 133399fee1e447aba4e74af5695a6dd5 [term 1 FOLLOWER]: Refusing update from remote peer 53b1888e4c9546a5af572aa5c87b29e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:32:28.135495 14291 raft_consensus.cc:1261] T 8248317c305d4f13b8c66cd548bb3072 P b611f7b8e9584eb4928bd774e1721703 [term 1 FOLLOWER]: Refusing update from remote peer 53b1888e4c9546a5af572aa5c87b29e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:32:28.136266 14633 consensus_queue.cc:1035] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "133399fee1e447aba4e74af5695a6dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:32:28.136507 14629 consensus_queue.cc:1035] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:32:28.144716 14638 mvcc.cc:205] Tried to move back new op lower bound from 7333569937955790848 to 7333569937714503680. Current Snapshot: MvccSnapshot[applied={T|T < 7333569937955790848}]
I20260926 11:32:28.144884 14637 mvcc.cc:205] Tried to move back new op lower bound from 7333569937955790848 to 7333569937714503680. Current Snapshot: MvccSnapshot[applied={T|T < 7333569937955790848}]
I20260926 11:32:28.153764 14146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:32:28.153962 14146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:32:28.156379 14146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7: Sending DeleteTablet for 3 replicas of tablet 8248317c305d4f13b8c66cd548bb3072
I20260926 11:32:28.157205 14405 tablet_service.cc:1514] Processing DeleteTablet for tablet 8248317c305d4f13b8c66cd548bb3072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:32:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37640
I20260926 11:32:28.157205 14271 tablet_service.cc:1514] Processing DeleteTablet for tablet 8248317c305d4f13b8c66cd548bb3072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:32:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34012
I20260926 11:32:28.157325 14539 tablet_service.cc:1514] Processing DeleteTablet for tablet 8248317c305d4f13b8c66cd548bb3072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:32:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49512
I20260926 11:32:28.157868 14647 tablet_replica.cc:330] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2: stopping tablet replica
I20260926 11:32:28.157874 14648 tablet_replica.cc:330] T 8248317c305d4f13b8c66cd548bb3072 P 133399fee1e447aba4e74af5695a6dd5: stopping tablet replica
I20260926 11:32:28.158021 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 14208
I20260926 11:32:28.158205 14648 raft_consensus.cc:2229] T 8248317c305d4f13b8c66cd548bb3072 P 133399fee1e447aba4e74af5695a6dd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:32:28.158205 14647 raft_consensus.cc:2229] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:32:28.158525 14648 raft_consensus.cc:2258] T 8248317c305d4f13b8c66cd548bb3072 P 133399fee1e447aba4e74af5695a6dd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:32:28.158620 14647 raft_consensus.cc:2258] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:32:28.159966 14647 ts_tablet_manager.cc:1905] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:32:28.160286 14648 ts_tablet_manager.cc:1905] T 8248317c305d4f13b8c66cd548bb3072 P 133399fee1e447aba4e74af5695a6dd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:32:28.163003 14647 ts_tablet_manager.cc:1918] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:32:28.163162 14647 log.cc:1196] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/wal/wals/8248317c305d4f13b8c66cd548bb3072
I20260926 11:32:28.163560 14647 ts_tablet_manager.cc:1939] T 8248317c305d4f13b8c66cd548bb3072 P 53b1888e4c9546a5af572aa5c87b29e2: Deleting consensus metadata
W20260926 11:32:28.164968 14132 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39175 (error 108)
I20260926 11:32:28.165390 14133 catalog_manager.cc:4841] TS 53b1888e4c9546a5af572aa5c87b29e2 (127.0.98.2:36243): tablet 8248317c305d4f13b8c66cd548bb3072 (table test-table [id=d72c2908348c4a2aa6c0ebb4460bc603]) successfully deleted
I20260926 11:32:28.165802 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 14342
I20260926 11:32:28.166348 14619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:28.166060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39175 (local address 127.0.0.1:34022)
0926 11:32:28.166210 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:28.166268 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260926 11:32:28.166505 14132 catalog_manager.cc:4584] TS b611f7b8e9584eb4928bd774e1721703 (127.0.98.1:39175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8248317c305d4f13b8c66cd548bb3072: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39175: connect: Connection refused (error 111)
I20260926 11:32:28.167285 14648 ts_tablet_manager.cc:1918] T 8248317c305d4f13b8c66cd548bb3072 P 133399fee1e447aba4e74af5695a6dd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:32:28.167378 14648 log.cc:1196] T 8248317c305d4f13b8c66cd548bb3072 P 133399fee1e447aba4e74af5695a6dd5: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/wal/wals/8248317c305d4f13b8c66cd548bb3072
I20260926 11:32:28.167692 14648 ts_tablet_manager.cc:1939] T 8248317c305d4f13b8c66cd548bb3072 P 133399fee1e447aba4e74af5695a6dd5: Deleting consensus metadata
I20260926 11:32:28.168844 14130 catalog_manager.cc:4841] TS 133399fee1e447aba4e74af5695a6dd5 (127.0.98.3:37853): tablet 8248317c305d4f13b8c66cd548bb3072 (table test-table [id=d72c2908348c4a2aa6c0ebb4460bc603]) successfully deleted
I20260926 11:32:28.171583 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 14476
I20260926 11:32:28.177526 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 14114
I20260926 11:32:28.184470 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36827
--webserver_interface=127.0.98.62
--webserver_port=41119
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:32:28.296936 14650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:32:28.297216 14650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:32:28.297299 14650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:32:28.301435 14650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:32:28.301496 14650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:32:28.301515 14650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:32:28.301532 14650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:32:28.301548 14650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:32:28.306989 14650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:32:28.308274 14650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:32:28.309324 14650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:32:28.322649 14659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:28.322642 14656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:28.322836 14650 server_base.cc:1034] running on GCE node
W20260926 11:32:28.322642 14657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:28.323536 14650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:32:28.324721 14650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:32:28.325886 14650 hybrid_clock.cc:648] HybridClock initialized: now 1790422348325867 us; error 32 us; skew 500 ppm
Sep 26 11:32:28 dist-test-slave-mrck krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:32:28.328796 14650 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:32:28.329736 14650 webserver.cc:460] Webserver started at http://127.0.98.62:41119/ using document root <none> and password file <none>
I20260926 11:32:28.330282 14650 fs_manager.cc:359] Metadata directory not provided
I20260926 11:32:28.330363 14650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:32:28.334553 14650 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 11:32:28.336959 14666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:28.338315 14650 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:32:28.338487 14650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "dda02a9ff30c4333b4e9c13a189efbb7"
format_stamp: "Formatted at 2026-09-26 11:32:26 on dist-test-slave-mrck"
server_key: "6c7b1994c5758f624b29c4c34662a7e2"
server_key_iv: "8d3e40550eeb1c57389c2d09306b8673"
server_key_version: "encryptionkey@0"
I20260926 11:32:28.339128 14650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:32:28.367146 14650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:32:28.370982 14650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:32:28.371199 14650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:32:28.378474 14650 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36827
I20260926 11:32:28.378753 14718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36827 every 8 connection(s)
I20260926 11:32:28.379640 14650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:32:28.380909 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 14650
I20260926 11:32:28.381371 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39175
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36827
--webserver_port=44801
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:32:28.384343 14719 sys_catalog.cc:263] Verifying existing consensus state
I20260926 11:32:28.386317 14719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7: Bootstrap starting.
I20260926 11:32:28.403447 14719 log.cc:824] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7: Log is configured to *not* fsync() on all Append() calls
I20260926 11:32:28.411945 14719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 11:32:28.412343 14719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7: Bootstrap complete.
I20260926 11:32:28.416920 14719 raft_consensus.cc:348] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda02a9ff30c4333b4e9c13a189efbb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36827 } }
I20260926 11:32:28.417309 14719 raft_consensus.cc:729] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dda02a9ff30c4333b4e9c13a189efbb7, State: Initialized, Role: FOLLOWER
I20260926 11:32:28.417848 14719 consensus_queue.cc:260] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda02a9ff30c4333b4e9c13a189efbb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36827 } }
I20260926 11:32:28.417972 14719 raft_consensus.cc:388] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:32:28.418064 14719 raft_consensus.cc:482] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:32:28.418162 14719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [term 1 FOLLOWER]: Advancing to term 2
I20260926 11:32:28.420136 14719 raft_consensus.cc:504] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda02a9ff30c4333b4e9c13a189efbb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36827 } }
I20260926 11:32:28.420468 14719 leader_election.cc:302] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dda02a9ff30c4333b4e9c13a189efbb7; no voters:
I20260926 11:32:28.420807 14719 leader_election.cc:288] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 11:32:28.420895 14723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 11:32:28.421093 14723 raft_consensus.cc:686] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [term 2 LEADER]: Becoming Leader. State: Replica: dda02a9ff30c4333b4e9c13a189efbb7, State: Running, Role: LEADER
I20260926 11:32:28.421454 14723 consensus_queue.cc:237] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda02a9ff30c4333b4e9c13a189efbb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36827 } }
I20260926 11:32:28.422066 14719 sys_catalog.cc:564] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:32:28.422935 14725 sys_catalog.cc:455] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dda02a9ff30c4333b4e9c13a189efbb7. Latest consensus state: current_term: 2 leader_uuid: "dda02a9ff30c4333b4e9c13a189efbb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda02a9ff30c4333b4e9c13a189efbb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36827 } } }
I20260926 11:32:28.422974 14724 sys_catalog.cc:455] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dda02a9ff30c4333b4e9c13a189efbb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda02a9ff30c4333b4e9c13a189efbb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36827 } } }
I20260926 11:32:28.423102 14724 sys_catalog.cc:458] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [sys.catalog]: This master's current role is: LEADER
I20260926 11:32:28.423056 14725 sys_catalog.cc:458] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [sys.catalog]: This master's current role is: LEADER
I20260926 11:32:28.423504 14731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:32:28.428442 14731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:32:28.429121 14731 catalog_manager.cc:1259] Loaded cluster ID: 2b2bd1c2a3be49eb92a27f7accacafe6
I20260926 11:32:28.429186 14731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:32:28.430546 14731 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 11:32:28.435472 14731 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 11:32:28.440769 14731 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:32:28.441514 14731 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7: Loaded TSK: 0
I20260926 11:32:28.442670 14731 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 11:32:28.498374 14721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:32:28.498692 14721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:32:28.498767 14721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:32:28.502871 14721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:32:28.502970 14721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:32:28.503064 14721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:32:28.508508 14721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44801
--tserver_master_addrs=127.0.98.62:36827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:32:28.509590 14721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:32:28.510658 14721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:32:28.521292 14746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:28.521399 14747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:28.521378 14749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:28.522023 14721 server_base.cc:1034] running on GCE node
I20260926 11:32:28.522558 14721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:32:28.523329 14721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:32:28.524531 14721 hybrid_clock.cc:648] HybridClock initialized: now 1790422348524505 us; error 38 us; skew 500 ppm
Sep 26 11:32:28 dist-test-slave-mrck krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:32:28.528110 14721 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:32:28.529351 14721 webserver.cc:460] Webserver started at http://127.0.98.1:44801/ using document root <none> and password file <none>
I20260926 11:32:28.530064 14721 fs_manager.cc:359] Metadata directory not provided
I20260926 11:32:28.530165 14721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:32:28.535411 14721 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260926 11:32:28.538179 14756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:28.539829 14721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:32:28.539963 14721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "b611f7b8e9584eb4928bd774e1721703"
format_stamp: "Formatted at 2026-09-26 11:32:27 on dist-test-slave-mrck"
server_key: "ab53f151e16d9bb0cf2a775b4cb576c8"
server_key_iv: "46b608683cee621e331c5681dfed117d"
server_key_version: "encryptionkey@0"
I20260926 11:32:28.540712 14721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:32:28.558733 14721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:32:28.562559 14721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:32:28.562752 14721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:32:28.563410 14721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:32:28.564725 14763 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 11:32:28.568902 14721 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 11:32:28.568972 14721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260926 11:32:28.569036 14721 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 11:32:28.571131 14721 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 11:32:28.571184 14721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 11:32:28.571528 14763 tablet_bootstrap.cc:492] T 8248317c305d4f13b8c66cd548bb3072 P b611f7b8e9584eb4928bd774e1721703: Bootstrap starting.
I20260926 11:32:28.581810 14721 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39175
I20260926 11:32:28.582067 14870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39175 every 8 connection(s)
I20260926 11:32:28.583105 14721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:32:28.588328 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 14721
I20260926 11:32:28.588960 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36243
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36827
--webserver_port=43129
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:32:28.592263 14763 log.cc:824] T 8248317c305d4f13b8c66cd548bb3072 P b611f7b8e9584eb4928bd774e1721703: Log is configured to *not* fsync() on all Append() calls
Sep 26 11:32:28 dist-test-slave-mrck krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:32:28.602280 14763 tablet_bootstrap.cc:492] T 8248317c305d4f13b8c66cd548bb3072 P b611f7b8e9584eb4928bd774e1721703: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 11:32:28.602543 14873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:28.585580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52687 (local address 127.0.98.62:36827)
0926 11:32:28.585927 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:28.585946 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:28.586452 (+ 506us) server_negotiation.cc:407] Connection header received
0926 11:32:28.588207 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:28.588223 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:28.588656 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:28.589029 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:28.590484 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:28.592017 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:28.593503 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:28.594033 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:28.594207 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:28.596740 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:28.596745 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:28.596756 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:28.596783 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:28.598655 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:28.599304 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:28.599309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:28.599314 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:28.599478 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:28.599927 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:28.599930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:28.599932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:28.600409 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:28.600521 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:28.601270 (+ 749us) server_negotiation.cc:299] Negotiation successful
0926 11:32:28.601614 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":126,"threads_started":1}
I20260926 11:32:28.603029 14874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:28.585563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.1:52687)
0926 11:32:28.586253 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:28.586312 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:28.587679 (+ 1367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:28.589324 (+ 1645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:28.589335 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:28.589783 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:32:28.590249 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:28.590264 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:28.592190 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:28.592195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:28.593363 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:28.593379 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:28.594177 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:28.594181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:28.594361 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:28.594991 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:32:28.595007 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:32:28.596547 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:32:28.598796 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:28.598802 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:28.598813 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:28.599170 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:28.599605 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:28.599611 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:28.599613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:28.599813 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:28.600609 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:32:28.600615 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:32:28.600919 (+ 304us) client_negotiation.cc:769] Sending connection context
0926 11:32:28.601874 (+ 955us) client_negotiation.cc:240] Negotiation successful
0926 11:32:28.602223 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":142,"threads_started":1}
I20260926 11:32:28.603235 14763 tablet_bootstrap.cc:492] T 8248317c305d4f13b8c66cd548bb3072 P b611f7b8e9584eb4928bd774e1721703: Bootstrap complete.
I20260926 11:32:28.604712 14763 ts_tablet_manager.cc:1397] T 8248317c305d4f13b8c66cd548bb3072 P b611f7b8e9584eb4928bd774e1721703: Time spent bootstrapping tablet: real 0.033s user 0.024s sys 0.007s
I20260926 11:32:28.605464 14871 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36827
I20260926 11:32:28.605690 14871 heartbeater.cc:464] Registering TS with master...
I20260926 11:32:28.606174 14871 heartbeater.cc:511] Master 127.0.98.62:36827 requested a full tablet report, sending...
I20260926 11:32:28.607779 14763 raft_consensus.cc:348] T 8248317c305d4f13b8c66cd548bb3072 P b611f7b8e9584eb4928bd774e1721703 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b1888e4c9546a5af572aa5c87b29e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36243 } } peers { permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 } } peers { permanent_uuid: "133399fee1e447aba4e74af5695a6dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37853 } }
I20260926 11:32:28.608316 14763 raft_consensus.cc:729] T 8248317c305d4f13b8c66cd548bb3072 P b611f7b8e9584eb4928bd774e1721703 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b611f7b8e9584eb4928bd774e1721703, State: Initialized, Role: FOLLOWER
I20260926 11:32:28.608637 14678 ts_manager.cc:194] Registered new tserver with Master: b611f7b8e9584eb4928bd774e1721703 (127.0.98.1:39175)
I20260926 11:32:28.608793 14763 consensus_queue.cc:260] T 8248317c305d4f13b8c66cd548bb3072 P b611f7b8e9584eb4928bd774e1721703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53b1888e4c9546a5af572aa5c87b29e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36243 } } peers { permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 } } peers { permanent_uuid: "133399fee1e447aba4e74af5695a6dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37853 } }
I20260926 11:32:28.609946 14763 ts_tablet_manager.cc:1428] T 8248317c305d4f13b8c66cd548bb3072 P b611f7b8e9584eb4928bd774e1721703: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260926 11:32:28.612185 14678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52687
I20260926 11:32:28.614498 14871 heartbeater.cc:503] Master 127.0.98.62:36827 was elected leader, sending a full tablet report...
Sep 26 11:32:28 dist-test-slave-mrck krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 11:32:28.623518 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:28.611629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39175 (local address 127.0.0.1:34032)
0926 11:32:28.612486 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:28.612531 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:28.612675 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:28.613190 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:28.613196 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:28.613643 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:32:28.613789 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:28.613799 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:28.616148 (+ 2349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:28.616153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:28.616915 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:28.616925 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:28.617495 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:28.617513 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:28.617624 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:28.618213 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:32:28.618225 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:32:28.619542 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:32:28.621518 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:28.621526 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:28.621531 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:28.621766 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:28.622124 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:28.622128 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:28.622130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:28.622218 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:28.622740 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:32:28.622747 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:32:28.622936 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 11:32:28.623107 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 11:32:28.623316 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":143,"threads_started":1}
I20260926 11:32:28.623525 14879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:28.611700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34032 (local address 127.0.98.1:39175)
0926 11:32:28.612069 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:28.612080 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:28.612634 (+ 554us) server_negotiation.cc:407] Connection header received
0926 11:32:28.612824 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:28.612829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:28.613028 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:28.613170 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:28.614937 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:28.615989 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:28.617036 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:28.617390 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:28.617506 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:28.619678 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:28.619687 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:28.619699 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:28.619782 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:28.621394 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:28.621933 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:28.621941 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:28.621943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:28.622032 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:28.622319 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:28.622322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:28.622324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:28.622618 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:28.622745 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:28.623205 (+ 460us) server_negotiation.cc:299] Negotiation successful
0926 11:32:28.623379 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":126,"threads_started":1}
I20260926 11:32:28.624600 14805 tablet_service.cc:1514] Processing DeleteTablet for tablet 8248317c305d4f13b8c66cd548bb3072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:32:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34032
I20260926 11:32:28.625465 14880 tablet_replica.cc:330] stopping tablet replica
I20260926 11:32:28.625803 14880 raft_consensus.cc:2229] T 8248317c305d4f13b8c66cd548bb3072 P b611f7b8e9584eb4928bd774e1721703 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:32:28.626067 14880 raft_consensus.cc:2258] T 8248317c305d4f13b8c66cd548bb3072 P b611f7b8e9584eb4928bd774e1721703 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:32:28.627009 14880 ts_tablet_manager.cc:1905] T 8248317c305d4f13b8c66cd548bb3072 P b611f7b8e9584eb4928bd774e1721703: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:32:28.630534 14880 ts_tablet_manager.cc:1918] T 8248317c305d4f13b8c66cd548bb3072 P b611f7b8e9584eb4928bd774e1721703: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:32:28.630615 14880 log.cc:1196] T 8248317c305d4f13b8c66cd548bb3072 P b611f7b8e9584eb4928bd774e1721703: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/wal/wals/8248317c305d4f13b8c66cd548bb3072
I20260926 11:32:28.630883 14880 ts_tablet_manager.cc:1939] T 8248317c305d4f13b8c66cd548bb3072 P b611f7b8e9584eb4928bd774e1721703: Deleting consensus metadata
I20260926 11:32:28.632723 14669 catalog_manager.cc:4841] TS b611f7b8e9584eb4928bd774e1721703 (127.0.98.1:39175): tablet 8248317c305d4f13b8c66cd548bb3072 (table test-table [id=d72c2908348c4a2aa6c0ebb4460bc603]) successfully deleted
W20260926 11:32:28.709880 14875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:32:28.710150 14875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:32:28.710192 14875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:32:28.715983 14875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:32:28.716070 14875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:32:28.716164 14875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:32:28.723423 14875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43129
--tserver_master_addrs=127.0.98.62:36827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:32:28.724599 14875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:32:28.725577 14875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:32:28.734871 14888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:28.734876 14885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:28.734871 14886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:28.735500 14875 server_base.cc:1034] running on GCE node
I20260926 11:32:28.735893 14875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:32:28.736479 14875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:32:28.737636 14875 hybrid_clock.cc:648] HybridClock initialized: now 1790422348737621 us; error 28 us; skew 500 ppm
Sep 26 11:32:28 dist-test-slave-mrck krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:32:28.740413 14875 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:32:28.741345 14875 webserver.cc:460] Webserver started at http://127.0.98.2:43129/ using document root <none> and password file <none>
I20260926 11:32:28.741930 14875 fs_manager.cc:359] Metadata directory not provided
I20260926 11:32:28.742033 14875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:32:28.746038 14875 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:32:28.748147 14895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:28.749512 14875 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:32:28.749647 14875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "53b1888e4c9546a5af572aa5c87b29e2"
format_stamp: "Formatted at 2026-09-26 11:32:27 on dist-test-slave-mrck"
server_key: "a99e397585403845e9ed6288843965f3"
server_key_iv: "51e708aff71c1d1fac7ec47c3e127ce9"
server_key_version: "encryptionkey@0"
I20260926 11:32:28.750198 14875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:32:28.761624 14875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:32:28.764464 14875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:32:28.764667 14875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:32:28.765259 14875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:32:28.766073 14875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:32:28.766145 14875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:28.766213 14875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:32:28.766260 14875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:28.775591 14875 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36243
I20260926 11:32:28.775612 15008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36243 every 8 connection(s)
I20260926 11:32:28.776677 14875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:32:28.776861 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 14875
I20260926 11:32:28.777395 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37853
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36827
--webserver_port=38563
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 11:32:28 dist-test-slave-mrck krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:32:28.793103 14873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:28.778787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52725 (local address 127.0.98.62:36827)
0926 11:32:28.778942 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:28.778948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:28.780307 (+ 1359us) server_negotiation.cc:407] Connection header received
0926 11:32:28.781667 (+ 1360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:28.781670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:28.781722 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:28.781789 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:28.783538 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:28.784094 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:28.786004 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:28.786238 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:28.786334 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:28.789283 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:28.789286 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:28.789288 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:28.789314 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:28.790713 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:28.791241 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:28.791245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:28.791247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:28.791327 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:28.791956 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:28.791963 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:28.791964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:28.792151 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:28.792285 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:28.792776 (+ 491us) server_negotiation.cc:299] Negotiation successful
0926 11:32:28.792913 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 11:32:28.793956 15012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:28.779208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.2:52725)
0926 11:32:28.780139 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:28.780188 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:28.781402 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:28.782013 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:28.782025 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:28.782612 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:32:28.783366 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:28.783389 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:28.784220 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:28.784227 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:28.785834 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:28.785849 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:28.786405 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:28.786412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:28.786669 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:28.787308 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:32:28.787331 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:32:28.789102 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:32:28.790844 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:28.790850 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:28.790866 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:28.791134 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:28.791429 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:28.791432 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:28.791434 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:28.791817 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:28.792262 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:32:28.792268 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:32:28.792535 (+ 267us) client_negotiation.cc:769] Sending connection context
0926 11:32:28.792721 (+ 186us) client_negotiation.cc:240] Negotiation successful
0926 11:32:28.793047 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":613,"threads_started":1}
I20260926 11:32:28.795197 15009 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36827
I20260926 11:32:28.795457 15009 heartbeater.cc:464] Registering TS with master...
I20260926 11:32:28.796066 15009 heartbeater.cc:511] Master 127.0.98.62:36827 requested a full tablet report, sending...
I20260926 11:32:28.797153 14678 ts_manager.cc:194] Registered new tserver with Master: 53b1888e4c9546a5af572aa5c87b29e2 (127.0.98.2:36243)
I20260926 11:32:28.797931 14678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52725
W20260926 11:32:28.894623 15011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:32:28.894887 15011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:32:28.894932 15011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:32:28.899017 15011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:32:28.899080 15011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:32:28.899139 15011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:32:28.904270 15011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38563
--tserver_master_addrs=127.0.98.62:36827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:32:28.905339 15011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:32:28.906265 15011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:32:28.915474 15020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:28.915548 15017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:28.915472 15018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:28.916495 15011 server_base.cc:1034] running on GCE node
I20260926 11:32:28.916893 15011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:32:28.917552 15011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:32:28.918710 15011 hybrid_clock.cc:648] HybridClock initialized: now 1790422348918696 us; error 27 us; skew 500 ppm
Sep 26 11:32:28 dist-test-slave-mrck krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:32:28.921559 15011 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:32:28.922470 15011 webserver.cc:460] Webserver started at http://127.0.98.3:38563/ using document root <none> and password file <none>
I20260926 11:32:28.923151 15011 fs_manager.cc:359] Metadata directory not provided
I20260926 11:32:28.923247 15011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:32:28.927322 15011 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 11:32:28.929522 15027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:28.930936 15011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 11:32:28.931044 15011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "133399fee1e447aba4e74af5695a6dd5"
format_stamp: "Formatted at 2026-09-26 11:32:27 on dist-test-slave-mrck"
server_key: "2faa3156c063d6d21e6fc434c83d08ed"
server_key_iv: "46589ce97cb51263d1ecd1b03224a41b"
server_key_version: "encryptionkey@0"
I20260926 11:32:28.931815 15011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:32:28.956375 15011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:32:28.959388 15011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:32:28.959563 15011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:32:28.960208 15011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:32:28.961022 15011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:32:28.961068 15011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:28.961107 15011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:32:28.961122 15011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:28.970120 15011 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37853
I20260926 11:32:28.970155 15140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37853 every 8 connection(s)
I20260926 11:32:28.971094 15011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:32:28.975644 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 15011
Sep 26 11:32:28 dist-test-slave-mrck krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422348, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:32:28.985044 15145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:28.976364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58804 (local address 127.0.98.62:36827)
0926 11:32:28.976611 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:28.976616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:28.976724 (+ 108us) server_negotiation.cc:407] Connection header received
0926 11:32:28.976855 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:28.976858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:28.976941 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:28.977037 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:28.977710 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:28.978213 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:28.979037 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:28.979264 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:28.979365 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:28.980775 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:28.980778 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:28.980781 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:28.980806 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:28.982792 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:28.983293 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:28.983297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:28.983298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:28.983382 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:28.983697 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:28.983700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:28.983702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:28.983949 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:28.984328 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:28.984704 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 11:32:28.984842 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":76,"threads_started":1}
I20260926 11:32:28.985944 14873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:28.972971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56253 (local address 127.0.98.62:36827)
0926 11:32:28.973115 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:28.973118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:28.974006 (+ 888us) server_negotiation.cc:407] Connection header received
0926 11:32:28.975305 (+ 1299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:28.975309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:28.975366 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:28.975453 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:28.977006 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:28.977540 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:28.978724 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:28.978963 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:28.979079 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:28.981868 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:28.981872 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:28.981874 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:28.981915 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:28.983707 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:28.984217 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:28.984220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:28.984221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:28.984318 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:28.984723 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:28.984728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:28.984730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:28.984898 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:28.985027 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:28.985685 (+ 658us) server_negotiation.cc:299] Negotiation successful
0926 11:32:28.985801 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 11:32:28.986956 15143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:28.973344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.3:56253)
0926 11:32:28.973837 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:28.973879 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:28.975092 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:28.975664 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:28.975671 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:28.976125 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:32:28.976825 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:28.976848 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:28.977677 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:28.977681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:28.978576 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:28.978585 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:28.979114 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:28.979118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:28.979290 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:28.979953 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:32:28.979980 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:32:28.981679 (+ 1699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:32:28.983850 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:28.983861 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:28.983874 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:28.984132 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:28.984442 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:28.984445 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:28.984446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:28.984605 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:28.985030 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:32:28.985039 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:32:28.985387 (+ 348us) client_negotiation.cc:769] Sending connection context
0926 11:32:28.985631 (+ 244us) client_negotiation.cc:240] Negotiation successful
0926 11:32:28.985978 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":136,"threads_started":1}
I20260926 11:32:28.988094 15141 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36827
I20260926 11:32:28.988359 15141 heartbeater.cc:464] Registering TS with master...
I20260926 11:32:28.988857 15141 heartbeater.cc:511] Master 127.0.98.62:36827 requested a full tablet report, sending...
I20260926 11:32:28.989900 14678 ts_manager.cc:194] Registered new tserver with Master: 133399fee1e447aba4e74af5695a6dd5 (127.0.98.3:37853)
I20260926 11:32:28.990490 14678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56253
I20260926 11:32:28.999163 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:32:29.003680 14873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:28.999832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58810 (local address 127.0.98.62:36827)
0926 11:32:28.999986 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:28.999991 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:29.000324 (+ 333us) server_negotiation.cc:407] Connection header received
0926 11:32:29.000408 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:29.000410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:29.000535 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:29.000625 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:32:29.000849 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:29.001347 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:29.002213 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:29.002470 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:29.002558 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:29.002832 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:32:29.003233 (+ 401us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:32:29.003325 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:29.003423 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 11:32:29.003512 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 11:32:29.004413 14678 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:58810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:32:29.007572 14678 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:32:29.015568 14805 tablet_service.cc:1467] Processing CreateTablet for tablet cbe3811fe91a4ec082cadd93d13cb307 (DEFAULT_TABLE table=test-table [id=5d0f53bd40ee4ca49ff287ecb76697cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:32:29.016297 14805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbe3811fe91a4ec082cadd93d13cb307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:32:29.019299 14763 tablet_bootstrap.cc:492] T cbe3811fe91a4ec082cadd93d13cb307 P b611f7b8e9584eb4928bd774e1721703: Bootstrap starting.
I20260926 11:32:29.020067 15151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:29.014778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36243 (local address 127.0.0.1:37658)
0926 11:32:29.015395 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:29.015412 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:29.015533 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:29.016060 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:29.016062 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:29.016081 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:32:29.016171 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:29.016181 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:29.017761 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:29.017765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:29.018495 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:29.018504 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:29.019553 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:29.019557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:29.019645 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:29.019690 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 11:32:29.019833 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 11:32:29.019912 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":99,"threads_started":1}
I20260926 11:32:29.020255 14763 tablet_bootstrap.cc:654] T cbe3811fe91a4ec082cadd93d13cb307 P b611f7b8e9584eb4928bd774e1721703: Neither blocks nor log segments found. Creating new log.
I20260926 11:32:29.020534 15152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:29.014979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37658 (local address 127.0.98.2:36243)
0926 11:32:29.015267 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:29.015281 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:29.015500 (+ 219us) server_negotiation.cc:407] Connection header received
0926 11:32:29.015682 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:29.015688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:29.015912 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:29.016067 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:32:29.016314 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:29.017628 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:29.018616 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:29.019210 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:29.019323 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:29.019475 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:29.020258 (+ 783us) server_negotiation.cc:299] Negotiation successful
0926 11:32:29.020367 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":110,"threads_started":1}
I20260926 11:32:29.020834 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:29.014352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37853 (local address 127.0.0.1:49528)
0926 11:32:29.014491 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:29.014516 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:29.014600 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:29.015575 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:29.015579 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:29.015600 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:32:29.015709 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:29.015715 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:29.017313 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:29.017317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:29.018536 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:29.018546 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:29.020446 (+ 1900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:29.020449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:29.020534 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:29.020552 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 11:32:29.020617 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 11:32:29.020705 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":4}
I20260926 11:32:29.021561 15150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:29.014536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49528 (local address 127.0.98.3:37853)
0926 11:32:29.015018 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:29.015030 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:29.015055 (+ 25us) server_negotiation.cc:407] Connection header received
0926 11:32:29.015135 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:29.015142 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:29.015372 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:29.015604 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:32:29.015874 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:29.017154 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:29.018683 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:29.020259 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:29.021125 (+ 866us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:29.021277 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:29.021330 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 11:32:29.021441 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":180,"threads_started":1}
I20260926 11:32:29.022132 14763 tablet_bootstrap.cc:492] T cbe3811fe91a4ec082cadd93d13cb307 P b611f7b8e9584eb4928bd774e1721703: No bootstrap required, opened a new log
I20260926 11:32:29.022222 14763 ts_tablet_manager.cc:1397] T cbe3811fe91a4ec082cadd93d13cb307 P b611f7b8e9584eb4928bd774e1721703: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 11:32:29.022672 14763 raft_consensus.cc:348] T cbe3811fe91a4ec082cadd93d13cb307 P b611f7b8e9584eb4928bd774e1721703 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133399fee1e447aba4e74af5695a6dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37853 } } peers { permanent_uuid: "53b1888e4c9546a5af572aa5c87b29e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36243 } } peers { permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 } }
I20260926 11:32:29.022816 14763 raft_consensus.cc:374] T cbe3811fe91a4ec082cadd93d13cb307 P b611f7b8e9584eb4928bd774e1721703 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:32:29.022855 14763 raft_consensus.cc:729] T cbe3811fe91a4ec082cadd93d13cb307 P b611f7b8e9584eb4928bd774e1721703 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b611f7b8e9584eb4928bd774e1721703, State: Initialized, Role: FOLLOWER
I20260926 11:32:29.023015 14763 consensus_queue.cc:260] T cbe3811fe91a4ec082cadd93d13cb307 P b611f7b8e9584eb4928bd774e1721703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133399fee1e447aba4e74af5695a6dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37853 } } peers { permanent_uuid: "53b1888e4c9546a5af572aa5c87b29e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36243 } } peers { permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 } }
I20260926 11:32:29.023334 14763 ts_tablet_manager.cc:1428] T cbe3811fe91a4ec082cadd93d13cb307 P b611f7b8e9584eb4928bd774e1721703: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 11:32:29.023813 14943 tablet_service.cc:1467] Processing CreateTablet for tablet cbe3811fe91a4ec082cadd93d13cb307 (DEFAULT_TABLE table=test-table [id=5d0f53bd40ee4ca49ff287ecb76697cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:32:29.024356 15075 tablet_service.cc:1467] Processing CreateTablet for tablet cbe3811fe91a4ec082cadd93d13cb307 (DEFAULT_TABLE table=test-table [id=5d0f53bd40ee4ca49ff287ecb76697cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:32:29.024868 14943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbe3811fe91a4ec082cadd93d13cb307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:32:29.025352 15075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbe3811fe91a4ec082cadd93d13cb307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:32:29.031446 15154 tablet_bootstrap.cc:492] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5: Bootstrap starting.
I20260926 11:32:29.032477 15155 tablet_bootstrap.cc:492] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2: Bootstrap starting.
I20260926 11:32:29.033717 15154 tablet_bootstrap.cc:654] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5: Neither blocks nor log segments found. Creating new log.
I20260926 11:32:29.034610 15155 tablet_bootstrap.cc:654] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2: Neither blocks nor log segments found. Creating new log.
I20260926 11:32:29.034840 15154 log.cc:824] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5: Log is configured to *not* fsync() on all Append() calls
I20260926 11:32:29.035459 15155 log.cc:824] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2: Log is configured to *not* fsync() on all Append() calls
I20260926 11:32:29.036957 15154 tablet_bootstrap.cc:492] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5: No bootstrap required, opened a new log
I20260926 11:32:29.037156 15154 ts_tablet_manager.cc:1397] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 11:32:29.037963 15155 tablet_bootstrap.cc:492] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2: No bootstrap required, opened a new log
I20260926 11:32:29.038156 15155 ts_tablet_manager.cc:1397] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 11:32:29.040390 15154 raft_consensus.cc:348] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133399fee1e447aba4e74af5695a6dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37853 } } peers { permanent_uuid: "53b1888e4c9546a5af572aa5c87b29e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36243 } } peers { permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 } }
I20260926 11:32:29.040663 15154 raft_consensus.cc:374] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:32:29.040742 15154 raft_consensus.cc:729] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133399fee1e447aba4e74af5695a6dd5, State: Initialized, Role: FOLLOWER
I20260926 11:32:29.041224 15154 consensus_queue.cc:260] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133399fee1e447aba4e74af5695a6dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37853 } } peers { permanent_uuid: "53b1888e4c9546a5af572aa5c87b29e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36243 } } peers { permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 } }
I20260926 11:32:29.041091 15155 raft_consensus.cc:348] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133399fee1e447aba4e74af5695a6dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37853 } } peers { permanent_uuid: "53b1888e4c9546a5af572aa5c87b29e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36243 } } peers { permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 } }
I20260926 11:32:29.041358 15155 raft_consensus.cc:374] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:32:29.041441 15155 raft_consensus.cc:729] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53b1888e4c9546a5af572aa5c87b29e2, State: Initialized, Role: FOLLOWER
I20260926 11:32:29.041884 15141 heartbeater.cc:503] Master 127.0.98.62:36827 was elected leader, sending a full tablet report...
I20260926 11:32:29.041864 15155 consensus_queue.cc:260] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133399fee1e447aba4e74af5695a6dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37853 } } peers { permanent_uuid: "53b1888e4c9546a5af572aa5c87b29e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36243 } } peers { permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 } }
I20260926 11:32:29.042258 15154 ts_tablet_manager.cc:1428] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260926 11:32:29.042361 15158 raft_consensus.cc:482] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:32:29.042490 15158 raft_consensus.cc:504] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133399fee1e447aba4e74af5695a6dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37853 } } peers { permanent_uuid: "53b1888e4c9546a5af572aa5c87b29e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36243 } } peers { permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 } }
I20260926 11:32:29.042636 15009 heartbeater.cc:503] Master 127.0.98.62:36827 was elected leader, sending a full tablet report...
I20260926 11:32:29.042860 15155 ts_tablet_manager.cc:1428] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 11:32:29.044132 15158 leader_election.cc:288] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53b1888e4c9546a5af572aa5c87b29e2 (127.0.98.2:36243), b611f7b8e9584eb4928bd774e1721703 (127.0.98.1:39175)
I20260926 11:32:29.048480 15152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:29.044379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37847 (local address 127.0.98.2:36243)
0926 11:32:29.044492 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:29.044496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:29.044510 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:32:29.044625 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:29.044630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:29.044676 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:29.044746 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:32:29.045130 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:29.045825 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:29.047253 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:29.047894 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:29.048018 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:29.048081 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:29.048334 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 11:32:29.048390 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 11:32:29.048504 15143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:29.044299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36243 (local address 127.0.98.3:37847)
0926 11:32:29.044423 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:29.044440 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:29.044537 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:29.044852 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:29.044856 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:29.044876 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:32:29.044988 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:29.044995 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:29.045917 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:29.045920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:29.047121 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:29.047130 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:29.048017 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:29.048020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:29.048186 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:29.048203 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 11:32:29.048298 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 11:32:29.048369 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":43}
I20260926 11:32:29.049082 14963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbe3811fe91a4ec082cadd93d13cb307" candidate_uuid: "133399fee1e447aba4e74af5695a6dd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53b1888e4c9546a5af572aa5c87b29e2" is_pre_election: true
I20260926 11:32:29.049383 14963 raft_consensus.cc:2454] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 133399fee1e447aba4e74af5695a6dd5 in term 0.
I20260926 11:32:29.049547 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:29.044569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39175 (local address 127.0.98.3:46413)
0926 11:32:29.045160 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:29.045174 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:29.045257 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:29.045793 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:29.045796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:29.045811 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:32:29.045959 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:29.045966 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:29.046975 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:29.046978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:29.048169 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:29.048178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:29.049129 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:29.049134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:29.049245 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:29.049269 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 11:32:29.049351 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 11:32:29.049400 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"mutex_wait_us":60,"thread_start_us":92,"threads_started":1}
I20260926 11:32:29.049566 14879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:29.044976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46413 (local address 127.0.98.1:39175)
0926 11:32:29.045230 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:29.045235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:29.045248 (+ 13us) server_negotiation.cc:407] Connection header received
0926 11:32:29.045408 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:29.045411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:29.045524 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:29.045608 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:32:29.046087 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:29.046866 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:29.048282 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:29.048980 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:29.049086 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:29.049238 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:29.049376 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 11:32:29.049455 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260926 11:32:29.049882 15031 leader_election.cc:302] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 133399fee1e447aba4e74af5695a6dd5, 53b1888e4c9546a5af572aa5c87b29e2; no voters:
I20260926 11:32:29.049993 14825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbe3811fe91a4ec082cadd93d13cb307" candidate_uuid: "133399fee1e447aba4e74af5695a6dd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b611f7b8e9584eb4928bd774e1721703" is_pre_election: true
I20260926 11:32:29.050150 15158 raft_consensus.cc:2781] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:32:29.050230 15158 raft_consensus.cc:482] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:32:29.050302 15158 raft_consensus.cc:3037] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:32:29.050302 14825 raft_consensus.cc:2454] T cbe3811fe91a4ec082cadd93d13cb307 P b611f7b8e9584eb4928bd774e1721703 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 133399fee1e447aba4e74af5695a6dd5 in term 0.
I20260926 11:32:29.051409 15158 raft_consensus.cc:504] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133399fee1e447aba4e74af5695a6dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37853 } } peers { permanent_uuid: "53b1888e4c9546a5af572aa5c87b29e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36243 } } peers { permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 } }
I20260926 11:32:29.051812 15158 leader_election.cc:288] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5 [CANDIDATE]: Term 1 election: Requested vote from peers 53b1888e4c9546a5af572aa5c87b29e2 (127.0.98.2:36243), b611f7b8e9584eb4928bd774e1721703 (127.0.98.1:39175)
I20260926 11:32:29.052083 14963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbe3811fe91a4ec082cadd93d13cb307" candidate_uuid: "133399fee1e447aba4e74af5695a6dd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53b1888e4c9546a5af572aa5c87b29e2"
I20260926 11:32:29.052220 14963 raft_consensus.cc:3037] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:32:29.052198 14825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbe3811fe91a4ec082cadd93d13cb307" candidate_uuid: "133399fee1e447aba4e74af5695a6dd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b611f7b8e9584eb4928bd774e1721703"
I20260926 11:32:29.052352 14825 raft_consensus.cc:3037] T cbe3811fe91a4ec082cadd93d13cb307 P b611f7b8e9584eb4928bd774e1721703 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:32:29.053355 14963 raft_consensus.cc:2454] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 133399fee1e447aba4e74af5695a6dd5 in term 1.
I20260926 11:32:29.053545 14825 raft_consensus.cc:2454] T cbe3811fe91a4ec082cadd93d13cb307 P b611f7b8e9584eb4928bd774e1721703 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 133399fee1e447aba4e74af5695a6dd5 in term 1.
I20260926 11:32:29.053794 15031 leader_election.cc:302] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 133399fee1e447aba4e74af5695a6dd5, 53b1888e4c9546a5af572aa5c87b29e2; no voters:
I20260926 11:32:29.054072 15158 raft_consensus.cc:2781] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:32:29.054170 15158 raft_consensus.cc:686] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5 [term 1 LEADER]: Becoming Leader. State: Replica: 133399fee1e447aba4e74af5695a6dd5, State: Running, Role: LEADER
I20260926 11:32:29.054497 15158 consensus_queue.cc:237] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133399fee1e447aba4e74af5695a6dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37853 } } peers { permanent_uuid: "53b1888e4c9546a5af572aa5c87b29e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36243 } } peers { permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 } }
I20260926 11:32:29.057668 14678 catalog_manager.cc:5495] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 133399fee1e447aba4e74af5695a6dd5 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "133399fee1e447aba4e74af5695a6dd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133399fee1e447aba4e74af5695a6dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53b1888e4c9546a5af572aa5c87b29e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 } health_report { overall_health: UNKNOWN } } }
I20260926 11:32:29.072770 15150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:29.069057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49540 (local address 127.0.98.3:37853)
0926 11:32:29.069180 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:29.069183 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:29.069196 (+ 13us) server_negotiation.cc:407] Connection header received
0926 11:32:29.069269 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:29.069272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:29.069328 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:29.069425 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:32:29.069645 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:29.070157 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:29.071043 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:29.071276 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:29.071369 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:29.071698 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:32:29.072299 (+ 601us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:32:29.072465 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:29.072577 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 11:32:29.072644 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 11:32:29.079303 14825 raft_consensus.cc:1261] T cbe3811fe91a4ec082cadd93d13cb307 P b611f7b8e9584eb4928bd774e1721703 [term 1 FOLLOWER]: Refusing update from remote peer 133399fee1e447aba4e74af5695a6dd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:32:29.079303 14963 raft_consensus.cc:1261] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2 [term 1 FOLLOWER]: Refusing update from remote peer 133399fee1e447aba4e74af5695a6dd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:32:29.080083 15159 consensus_queue.cc:1035] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53b1888e4c9546a5af572aa5c87b29e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:32:29.080300 15158 consensus_queue.cc:1035] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 11:32:29.084553 14872 tablet.cc:2369] T cbe3811fe91a4ec082cadd93d13cb307 P b611f7b8e9584eb4928bd774e1721703: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:32:29.087042 15165 mvcc.cc:205] Tried to move back new op lower bound from 7333569941821202432 to 7333569941730095104. Current Snapshot: MvccSnapshot[applied={T|T < 7333569941821202432}]
I20260926 11:32:29.089195 15166 mvcc.cc:205] Tried to move back new op lower bound from 7333569941821202432 to 7333569941730095104. Current Snapshot: MvccSnapshot[applied={T|T < 7333569941821202432}]
I20260926 11:32:29.095114 14679 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:58810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:32:29.095287 14679 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:58810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:32:29.097734 14679 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7: Sending DeleteTablet for 3 replicas of tablet cbe3811fe91a4ec082cadd93d13cb307
I20260926 11:32:29.098385 15075 tablet_service.cc:1514] Processing DeleteTablet for tablet cbe3811fe91a4ec082cadd93d13cb307 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:32:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49528
I20260926 11:32:29.098515 14943 tablet_service.cc:1514] Processing DeleteTablet for tablet cbe3811fe91a4ec082cadd93d13cb307 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:32:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37658
I20260926 11:32:29.098887 15174 tablet_replica.cc:330] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5: stopping tablet replica
I20260926 11:32:29.099041 14805 tablet_service.cc:1514] Processing DeleteTablet for tablet cbe3811fe91a4ec082cadd93d13cb307 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:32:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34032
I20260926 11:32:29.099076 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 14721
I20260926 11:32:29.099845 15174 raft_consensus.cc:2229] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:32:29.100224 15174 raft_consensus.cc:2258] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:32:29.101545 15174 ts_tablet_manager.cc:1905] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:32:29.104152 15174 ts_tablet_manager.cc:1918] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:32:29.104276 15174 log.cc:1196] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/wal/wals/cbe3811fe91a4ec082cadd93d13cb307
I20260926 11:32:29.104586 15174 ts_tablet_manager.cc:1939] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5: Deleting consensus metadata
I20260926 11:32:29.105662 14667 catalog_manager.cc:4841] TS 133399fee1e447aba4e74af5695a6dd5 (127.0.98.3:37853): tablet cbe3811fe91a4ec082cadd93d13cb307 (table test-table [id=5d0f53bd40ee4ca49ff287ecb76697cc]) successfully deleted
I20260926 11:32:29.108307 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 14875
W20260926 11:32:29.108340 14669 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39175 (error 108)
I20260926 11:32:29.110807 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:29.110537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39175 (local address 127.0.0.1:34044)
0926 11:32:29.110664 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:29.110736 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39175: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 11:32:29.111025 14669 catalog_manager.cc:4584] TS b611f7b8e9584eb4928bd774e1721703 (127.0.98.1:39175): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cbe3811fe91a4ec082cadd93d13cb307: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39175: connect: Connection refused (error 111)
I20260926 11:32:29.115367 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 15011
I20260926 11:32:29.116766 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:29.116304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36243 (local address 127.0.0.1:37672)
0926 11:32:29.116454 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:29.116641 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 11:32:29.121214 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 14650
I20260926 11:32:29.127523 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36827
--webserver_interface=127.0.98.62
--webserver_port=41119
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:32:29.238406 15177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:32:29.238740 15177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:32:29.238826 15177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:32:29.242931 15177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:32:29.243046 15177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:32:29.243088 15177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:32:29.243127 15177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:32:29.243163 15177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:32:29.248966 15177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41119
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:32:29.250219 15177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:32:29.251279 15177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:32:29.259936 15185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:29.259958 15183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:29.259934 15182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:29.260363 15177 server_base.cc:1034] running on GCE node
I20260926 11:32:29.260947 15177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:32:29.262034 15177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:32:29.263191 15177 hybrid_clock.cc:648] HybridClock initialized: now 1790422349263155 us; error 51 us; skew 500 ppm
Sep 26 11:32:29 dist-test-slave-mrck krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:32:29.266047 15177 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:32:29.266979 15177 webserver.cc:460] Webserver started at http://127.0.98.62:41119/ using document root <none> and password file <none>
I20260926 11:32:29.267504 15177 fs_manager.cc:359] Metadata directory not provided
I20260926 11:32:29.267563 15177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:32:29.271442 15177 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:32:29.273659 15192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:29.274962 15177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:32:29.275206 15177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "dda02a9ff30c4333b4e9c13a189efbb7"
format_stamp: "Formatted at 2026-09-26 11:32:26 on dist-test-slave-mrck"
server_key: "6c7b1994c5758f624b29c4c34662a7e2"
server_key_iv: "8d3e40550eeb1c57389c2d09306b8673"
server_key_version: "encryptionkey@0"
I20260926 11:32:29.275815 15177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:32:29.307968 15177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:32:29.311630 15177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:32:29.311887 15177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:32:29.319476 15177 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36827
I20260926 11:32:29.319810 15244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36827 every 8 connection(s)
I20260926 11:32:29.320672 15177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:32:29.323871 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 15177
I20260926 11:32:29.324330 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39175
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36827
--webserver_port=44801
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:32:29.325271 15245 sys_catalog.cc:263] Verifying existing consensus state
I20260926 11:32:29.327056 15245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7: Bootstrap starting.
I20260926 11:32:29.344496 15245 log.cc:824] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7: Log is configured to *not* fsync() on all Append() calls
I20260926 11:32:29.356084 15245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 11:32:29.356508 15245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7: Bootstrap complete.
I20260926 11:32:29.360603 15245 raft_consensus.cc:348] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda02a9ff30c4333b4e9c13a189efbb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36827 } }
I20260926 11:32:29.361011 15245 raft_consensus.cc:729] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: dda02a9ff30c4333b4e9c13a189efbb7, State: Initialized, Role: FOLLOWER
I20260926 11:32:29.361493 15245 consensus_queue.cc:260] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda02a9ff30c4333b4e9c13a189efbb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36827 } }
I20260926 11:32:29.361646 15245 raft_consensus.cc:388] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:32:29.361732 15245 raft_consensus.cc:482] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:32:29.361848 15245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [term 2 FOLLOWER]: Advancing to term 3
I20260926 11:32:29.363890 15245 raft_consensus.cc:504] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda02a9ff30c4333b4e9c13a189efbb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36827 } }
I20260926 11:32:29.364228 15245 leader_election.cc:302] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dda02a9ff30c4333b4e9c13a189efbb7; no voters:
I20260926 11:32:29.364578 15245 leader_election.cc:288] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 11:32:29.364657 15249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 11:32:29.364843 15249 raft_consensus.cc:686] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [term 3 LEADER]: Becoming Leader. State: Replica: dda02a9ff30c4333b4e9c13a189efbb7, State: Running, Role: LEADER
I20260926 11:32:29.365147 15249 consensus_queue.cc:237] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda02a9ff30c4333b4e9c13a189efbb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36827 } }
I20260926 11:32:29.365839 15245 sys_catalog.cc:564] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:32:29.366091 15249 sys_catalog.cc:455] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dda02a9ff30c4333b4e9c13a189efbb7. Latest consensus state: current_term: 3 leader_uuid: "dda02a9ff30c4333b4e9c13a189efbb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda02a9ff30c4333b4e9c13a189efbb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36827 } } }
I20260926 11:32:29.366209 15249 sys_catalog.cc:458] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [sys.catalog]: This master's current role is: LEADER
I20260926 11:32:29.366550 15253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:32:29.366645 15250 sys_catalog.cc:455] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "dda02a9ff30c4333b4e9c13a189efbb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dda02a9ff30c4333b4e9c13a189efbb7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36827 } } }
I20260926 11:32:29.366815 15250 sys_catalog.cc:458] T 00000000000000000000000000000000 P dda02a9ff30c4333b4e9c13a189efbb7 [sys.catalog]: This master's current role is: LEADER
I20260926 11:32:29.371762 15253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:32:29.372581 15253 catalog_manager.cc:1259] Loaded cluster ID: 2b2bd1c2a3be49eb92a27f7accacafe6
I20260926 11:32:29.372646 15253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 11:32:29.373672 15253 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7f924909ad google::LogMessage::Fail() at ??:0
@ 0x7f7f9249388a google::LogMessage::SendToLog() at ??:0
@ 0x7f7f924904de google::LogMessage::Flush() at ??:0
@ 0x7f7f92491a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7f9cdcffdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f7f9cdd0d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7f9cdcd579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f7f9ce11640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f7f946e23d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7f946e2c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f7f946e43f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f7f946d3512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7f95d176db start_thread at ??:0
@ 0x7f7f90f8e61f clone at ??:0
W20260926 11:32:29.466173 15247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:32:29.466571 15247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:32:29.466641 15247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:32:29.472980 15247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:32:29.473068 15247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:32:29.473163 15247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:32:29.481352 15247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44801
--tserver_master_addrs=127.0.98.62:36827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:32:29.482784 15247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:32:29.484103 15247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:32:29.495116 15286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:29.495127 15287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:29.495270 15247 server_base.cc:1034] running on GCE node
W20260926 11:32:29.495105 15289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:29.496009 15247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:32:29.496649 15247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:32:29.497835 15247 hybrid_clock.cc:648] HybridClock initialized: now 1790422349497817 us; error 39 us; skew 500 ppm
Sep 26 11:32:29 dist-test-slave-mrck krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:32:29.500665 15247 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:32:29.501595 15247 webserver.cc:460] Webserver started at http://127.0.98.1:44801/ using document root <none> and password file <none>
I20260926 11:32:29.502151 15247 fs_manager.cc:359] Metadata directory not provided
I20260926 11:32:29.502213 15247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:32:29.506150 15247 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 11:32:29.508355 15296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:29.509699 15247 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:32:29.509836 15247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "b611f7b8e9584eb4928bd774e1721703"
format_stamp: "Formatted at 2026-09-26 11:32:27 on dist-test-slave-mrck"
server_key: "ab53f151e16d9bb0cf2a775b4cb576c8"
server_key_iv: "46b608683cee621e331c5681dfed117d"
server_key_version: "encryptionkey@0"
I20260926 11:32:29.510393 15247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:32:29.545764 15247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:32:29.549984 15247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:32:29.550195 15247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:32:29.550982 15247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:32:29.552372 15303 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 11:32:29.556677 15247 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 11:32:29.556769 15247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260926 11:32:29.556841 15247 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 11:32:29.559136 15247 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 11:32:29.559196 15247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 11:32:29.559514 15303 tablet_bootstrap.cc:492] T cbe3811fe91a4ec082cadd93d13cb307 P b611f7b8e9584eb4928bd774e1721703: Bootstrap starting.
I20260926 11:32:29.570567 15247 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39175
I20260926 11:32:29.570719 15410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39175 every 8 connection(s)
I20260926 11:32:29.571835 15247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:32:29.579234 15303 log.cc:824] T cbe3811fe91a4ec082cadd93d13cb307 P b611f7b8e9584eb4928bd774e1721703: Log is configured to *not* fsync() on all Append() calls
I20260926 11:32:29.581972 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 15247
I20260926 11:32:29.582417 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36243
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36827
--webserver_port=43129
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:32:29.586637 15303 tablet_bootstrap.cc:492] T cbe3811fe91a4ec082cadd93d13cb307 P b611f7b8e9584eb4928bd774e1721703: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 11:32:29.587021 15303 tablet_bootstrap.cc:492] T cbe3811fe91a4ec082cadd93d13cb307 P b611f7b8e9584eb4928bd774e1721703: Bootstrap complete.
I20260926 11:32:29.588193 15303 ts_tablet_manager.cc:1397] T cbe3811fe91a4ec082cadd93d13cb307 P b611f7b8e9584eb4928bd774e1721703: Time spent bootstrapping tablet: real 0.029s user 0.017s sys 0.011s
I20260926 11:32:29.590590 15303 raft_consensus.cc:348] T cbe3811fe91a4ec082cadd93d13cb307 P b611f7b8e9584eb4928bd774e1721703 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133399fee1e447aba4e74af5695a6dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37853 } } peers { permanent_uuid: "53b1888e4c9546a5af572aa5c87b29e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36243 } } peers { permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 } }
I20260926 11:32:29.591018 15303 raft_consensus.cc:729] T cbe3811fe91a4ec082cadd93d13cb307 P b611f7b8e9584eb4928bd774e1721703 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b611f7b8e9584eb4928bd774e1721703, State: Initialized, Role: FOLLOWER
I20260926 11:32:29.591500 15303 consensus_queue.cc:260] T cbe3811fe91a4ec082cadd93d13cb307 P b611f7b8e9584eb4928bd774e1721703 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133399fee1e447aba4e74af5695a6dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37853 } } peers { permanent_uuid: "53b1888e4c9546a5af572aa5c87b29e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36243 } } peers { permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 } }
I20260926 11:32:29.592480 15303 ts_tablet_manager.cc:1428] T cbe3811fe91a4ec082cadd93d13cb307 P b611f7b8e9584eb4928bd774e1721703: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
W20260926 11:32:29.637719 15413 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:32:29.574101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.1:52259)
0926 11:32:29.574754 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:29.574819 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:29.576001 (+ 1182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:29.636962 (+ 60961us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":159,"threads_started":1}
W20260926 11:32:29.638500 15411 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36827 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36827: Client connection negotiation failed: client connection to 127.0.98.62:36827: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 11:32:29.639361 15413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:29.639005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.1:54057)
0926 11:32:29.639199 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:29.639289 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 11:32:29.640115 15413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:29.639886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.1:43921)
0926 11:32:29.639994 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:29.640045 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 11:32:29.640295 15411 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 11:32:29.711464 15415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:32:29.711805 15415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:32:29.711901 15415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:32:29.716079 15415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:32:29.716143 15415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:32:29.716243 15415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:32:29.721805 15415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36243
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=43129
--tserver_master_addrs=127.0.98.62:36827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:32:29.722920 15415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:32:29.723980 15415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:32:29.733675 15424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:29.733693 15421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:29.733697 15422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:29.734107 15415 server_base.cc:1034] running on GCE node
I20260926 11:32:29.734560 15415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:32:29.735250 15415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:32:29.736429 15415 hybrid_clock.cc:648] HybridClock initialized: now 1790422349736410 us; error 34 us; skew 500 ppm
Sep 26 11:32:29 dist-test-slave-mrck krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:32:29.739215 15415 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:32:29.740216 15415 webserver.cc:460] Webserver started at http://127.0.98.2:43129/ using document root <none> and password file <none>
I20260926 11:32:29.740784 15415 fs_manager.cc:359] Metadata directory not provided
I20260926 11:32:29.740877 15415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:32:29.744755 15415 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:32:29.746887 15431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:29.748235 15415 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:32:29.748381 15415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "53b1888e4c9546a5af572aa5c87b29e2"
format_stamp: "Formatted at 2026-09-26 11:32:27 on dist-test-slave-mrck"
server_key: "a99e397585403845e9ed6288843965f3"
server_key_iv: "51e708aff71c1d1fac7ec47c3e127ce9"
server_key_version: "encryptionkey@0"
I20260926 11:32:29.748952 15415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:32:29.768203 15415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:32:29.771130 15415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:32:29.771334 15415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:32:29.771991 15415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:32:29.773046 15438 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 11:32:29.777427 15415 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 11:32:29.777519 15415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260926 11:32:29.777606 15415 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 11:32:29.779214 15415 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 11:32:29.779304 15415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 11:32:29.779574 15438 tablet_bootstrap.cc:492] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2: Bootstrap starting.
I20260926 11:32:29.789906 15415 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36243
I20260926 11:32:29.790019 15545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36243 every 8 connection(s)
I20260926 11:32:29.790992 15415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:32:29.791848 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 15415
I20260926 11:32:29.792359 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37853
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36827
--webserver_port=38563
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:32:29.794528 15548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:29.793108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.2:46141)
0926 11:32:29.793614 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:29.793746 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":159,"threads_started":1}
W20260926 11:32:29.794981 15546 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36827 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36827: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
I20260926 11:32:29.795578 15548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:29.795344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.2:47777)
0926 11:32:29.795462 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:29.795521 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 11:32:29.796195 15548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:29.795992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.2:52539)
0926 11:32:29.796093 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:29.796139 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 11:32:29.796373 15546 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 11:32:29.799791 15438 log.cc:824] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2: Log is configured to *not* fsync() on all Append() calls
I20260926 11:32:29.807255 15438 tablet_bootstrap.cc:492] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 11:32:29.807658 15438 tablet_bootstrap.cc:492] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2: Bootstrap complete.
I20260926 11:32:29.808987 15438 ts_tablet_manager.cc:1397] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2: Time spent bootstrapping tablet: real 0.030s user 0.028s sys 0.000s
I20260926 11:32:29.812014 15438 raft_consensus.cc:348] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133399fee1e447aba4e74af5695a6dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37853 } } peers { permanent_uuid: "53b1888e4c9546a5af572aa5c87b29e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36243 } } peers { permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 } }
I20260926 11:32:29.812500 15438 raft_consensus.cc:729] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53b1888e4c9546a5af572aa5c87b29e2, State: Initialized, Role: FOLLOWER
I20260926 11:32:29.812983 15438 consensus_queue.cc:260] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133399fee1e447aba4e74af5695a6dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37853 } } peers { permanent_uuid: "53b1888e4c9546a5af572aa5c87b29e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36243 } } peers { permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 } }
I20260926 11:32:29.814116 15438 ts_tablet_manager.cc:1428] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 11:32:29.814337 15548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:29.814035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.2:56653)
0926 11:32:29.814183 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:29.814250 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 11:32:29.815143 15548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:29.814866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.2:45647)
0926 11:32:29.815011 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:29.815071 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 11:32:29.908882 15549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:32:29.909147 15549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:32:29.909191 15549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:32:29.913244 15549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:32:29.913303 15549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:32:29.913393 15549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:32:29.918903 15549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38563
--tserver_master_addrs=127.0.98.62:36827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:32:29.920002 15549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:32:29.921030 15549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:32:29.930727 15557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:29.930819 15549 server_base.cc:1034] running on GCE node
W20260926 11:32:29.930658 15559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:29.930658 15556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:29.931396 15549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:32:29.932119 15549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:32:29.933300 15549 hybrid_clock.cc:648] HybridClock initialized: now 1790422349933285 us; error 31 us; skew 500 ppm
Sep 26 11:32:29 dist-test-slave-mrck krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:32:29.936017 15549 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:32:29.936913 15549 webserver.cc:460] Webserver started at http://127.0.98.3:38563/ using document root <none> and password file <none>
I20260926 11:32:29.937455 15549 fs_manager.cc:359] Metadata directory not provided
I20260926 11:32:29.937546 15549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:32:29.941327 15549 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260926 11:32:29.943285 15566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:29.944572 15549 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:32:29.944717 15549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "133399fee1e447aba4e74af5695a6dd5"
format_stamp: "Formatted at 2026-09-26 11:32:27 on dist-test-slave-mrck"
server_key: "2faa3156c063d6d21e6fc434c83d08ed"
server_key_iv: "46589ce97cb51263d1ecd1b03224a41b"
server_key_version: "encryptionkey@0"
I20260926 11:32:29.945494 15549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:32:29.966133 15549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:32:29.969242 15549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:32:29.969473 15549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:32:29.970098 15549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:32:29.971046 15549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:32:29.971122 15549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:29.971202 15549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:32:29.971252 15549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:29.980999 15549 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37853
I20260926 11:32:29.981021 15679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37853 every 8 connection(s)
I20260926 11:32:29.982074 15549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:32:29.984220 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 15549
I20260926 11:32:29.986198 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:29.984288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.3:32955)
0926 11:32:29.984792 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:29.984925 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":136,"threads_started":1}
W20260926 11:32:29.986680 15680 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36827 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36827: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
I20260926 11:32:29.987257 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:29.987032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.3:50679)
0926 11:32:29.987158 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:29.987204 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 11:32:29.987901 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:29.987673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.3:58529)
0926 11:32:29.987816 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:29.987856 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 11:32:29.988087 15680 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 11:32:30.641352 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:30.640922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.1:52587)
0926 11:32:30.641187 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:30.641245 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":97,"threads_started":1}
I20260926 11:32:30.816468 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:30.816022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.2:59983)
0926 11:32:30.816302 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:30.816367 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":72,"threads_started":1}
I20260926 11:32:30.989063 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:30.988650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.3:49289)
0926 11:32:30.988891 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:30.988951 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":70,"threads_started":1}
I20260926 11:32:31.002813 15687 raft_consensus.cc:482] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:32:31.003007 15687 raft_consensus.cc:504] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133399fee1e447aba4e74af5695a6dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37853 } } peers { permanent_uuid: "53b1888e4c9546a5af572aa5c87b29e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36243 } } peers { permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 } }
I20260926 11:32:31.004279 15687 leader_election.cc:288] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 133399fee1e447aba4e74af5695a6dd5 (127.0.98.3:37853), b611f7b8e9584eb4928bd774e1721703 (127.0.98.1:39175)
Sep 26 11:32:31 dist-test-slave-mrck krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 26 11:32:31 dist-test-slave-mrck krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 11:32:31.020102 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:31.004502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39175 (local address 127.0.98.2:46687)
0926 11:32:31.005361 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:31.005377 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:31.005473 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:31.006217 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:31.006224 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:31.006606 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:32:31.007243 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:31.007264 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:31.009218 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:31.009222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:31.010986 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:31.010999 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:31.012027 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:31.012031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:31.012367 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:31.013156 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:32:31.013177 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:32:31.015454 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:32:31.017682 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:31.017685 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:31.017686 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:31.017954 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:31.018336 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:31.018338 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:31.018339 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:31.018449 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:31.019128 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:32:31.019131 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:32:31.019324 (+ 193us) client_negotiation.cc:769] Sending connection context
0926 11:32:31.019479 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 11:32:31.019892 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"mutex_wait_us":818,"spinlock_wait_cycles":28032,"thread_start_us":77,"threads_started":1}
I20260926 11:32:31.020103 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:31.004729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47501 (local address 127.0.98.3:37853)
0926 11:32:31.005052 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:31.005078 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:31.005123 (+ 45us) server_negotiation.cc:407] Connection header received
0926 11:32:31.006314 (+ 1191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:31.006321 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:31.006693 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:31.006917 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:31.007451 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:31.009029 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:31.011122 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:31.011884 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:31.012060 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:31.015244 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:31.015250 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:31.015267 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:31.015301 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:31.017444 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:31.018037 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:31.018045 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:31.018051 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:31.018180 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:31.018580 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:31.018583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:31.018584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:31.018971 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:31.019102 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:31.019643 (+ 541us) server_negotiation.cc:299] Negotiation successful
0926 11:32:31.019911 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":106,"threads_started":1}
I20260926 11:32:31.020113 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:31.004786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46687 (local address 127.0.98.1:39175)
0926 11:32:31.005245 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:31.005254 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:31.005472 (+ 218us) server_negotiation.cc:407] Connection header received
0926 11:32:31.005642 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:31.005649 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:31.005925 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:31.006122 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:31.007429 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:31.009029 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:31.011122 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:31.011884 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:31.012060 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:31.015629 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:31.015636 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:31.015652 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:31.015684 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:31.017562 (+ 1878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:31.018085 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:31.018089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:31.018093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:31.018204 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:31.018545 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:31.018551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:31.018553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:31.018982 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:31.019116 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:31.019626 (+ 510us) server_negotiation.cc:299] Negotiation successful
0926 11:32:31.019922 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":73,"threads_started":1}
I20260926 11:32:31.020131 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:31.004489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37853 (local address 127.0.98.2:47501)
0926 11:32:31.004701 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:31.004737 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:31.005227 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:31.006911 (+ 1684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:31.006914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:31.007102 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:32:31.007243 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:31.007264 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:31.009218 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:31.009222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:31.010986 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:31.010999 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:31.012027 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:31.012031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:31.012368 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:31.013147 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:32:31.013172 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:32:31.015020 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:32:31.017567 (+ 2547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:31.017574 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:31.017588 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:31.017923 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:31.018290 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:31.018294 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:31.018299 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:31.018462 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:31.019098 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:32:31.019104 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:32:31.019326 (+ 222us) client_negotiation.cc:769] Sending connection context
0926 11:32:31.019517 (+ 191us) client_negotiation.cc:240] Negotiation successful
0926 11:32:31.019972 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":445,"spinlock_wait_cycles":28288}
I20260926 11:32:31.021950 15365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbe3811fe91a4ec082cadd93d13cb307" candidate_uuid: "53b1888e4c9546a5af572aa5c87b29e2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b611f7b8e9584eb4928bd774e1721703" is_pre_election: true
I20260926 11:32:31.022264 15365 raft_consensus.cc:2454] T cbe3811fe91a4ec082cadd93d13cb307 P b611f7b8e9584eb4928bd774e1721703 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 53b1888e4c9546a5af572aa5c87b29e2 in term 1.
I20260926 11:32:31.023236 15434 leader_election.cc:302] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53b1888e4c9546a5af572aa5c87b29e2, b611f7b8e9584eb4928bd774e1721703; no voters:
I20260926 11:32:31.023567 15687 raft_consensus.cc:2781] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 11:32:31.023654 15687 raft_consensus.cc:482] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:32:31.023687 15687 raft_consensus.cc:3037] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2 [term 1 FOLLOWER]: Advancing to term 2
W20260926 11:32:31.024102 15432 leader_election.cc:341] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 133399fee1e447aba4e74af5695a6dd5 (127.0.98.3:37853): Not found: Tablet not found: cbe3811fe91a4ec082cadd93d13cb307
I20260926 11:32:31.025910 15687 raft_consensus.cc:504] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133399fee1e447aba4e74af5695a6dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37853 } } peers { permanent_uuid: "53b1888e4c9546a5af572aa5c87b29e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36243 } } peers { permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 } }
I20260926 11:32:31.026327 15687 leader_election.cc:288] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2 [CANDIDATE]: Term 2 election: Requested vote from peers 133399fee1e447aba4e74af5695a6dd5 (127.0.98.3:37853), b611f7b8e9584eb4928bd774e1721703 (127.0.98.1:39175)
W20260926 11:32:31.026933 15432 leader_election.cc:341] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 133399fee1e447aba4e74af5695a6dd5 (127.0.98.3:37853): Not found: Tablet not found: cbe3811fe91a4ec082cadd93d13cb307
I20260926 11:32:31.026981 15365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbe3811fe91a4ec082cadd93d13cb307" candidate_uuid: "53b1888e4c9546a5af572aa5c87b29e2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b611f7b8e9584eb4928bd774e1721703"
I20260926 11:32:31.027148 15365 raft_consensus.cc:3037] T cbe3811fe91a4ec082cadd93d13cb307 P b611f7b8e9584eb4928bd774e1721703 [term 1 FOLLOWER]: Advancing to term 2
I20260926 11:32:31.029246 15365 raft_consensus.cc:2454] T cbe3811fe91a4ec082cadd93d13cb307 P b611f7b8e9584eb4928bd774e1721703 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 53b1888e4c9546a5af572aa5c87b29e2 in term 2.
I20260926 11:32:31.029589 15434 leader_election.cc:302] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 53b1888e4c9546a5af572aa5c87b29e2, b611f7b8e9584eb4928bd774e1721703; no voters: 133399fee1e447aba4e74af5695a6dd5
I20260926 11:32:31.029775 15687 raft_consensus.cc:2781] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 11:32:31.029973 15687 raft_consensus.cc:686] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2 [term 2 LEADER]: Becoming Leader. State: Replica: 53b1888e4c9546a5af572aa5c87b29e2, State: Running, Role: LEADER
I20260926 11:32:31.030310 15687 consensus_queue.cc:237] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133399fee1e447aba4e74af5695a6dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37853 } } peers { permanent_uuid: "53b1888e4c9546a5af572aa5c87b29e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36243 } } peers { permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 } }
I20260926 11:32:31.030823 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:31.030491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.2:47947)
0926 11:32:31.030706 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:31.030757 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
W20260926 11:32:31.411801 15432 consensus_peers.cc:487] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2 -> Peer 133399fee1e447aba4e74af5695a6dd5 (127.0.98.3:37853): Couldn't send request to peer 133399fee1e447aba4e74af5695a6dd5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cbe3811fe91a4ec082cadd93d13cb307. This is attempt 1: this message will repeat every 5th retry.
I20260926 11:32:31.552214 15365 raft_consensus.cc:1261] T cbe3811fe91a4ec082cadd93d13cb307 P b611f7b8e9584eb4928bd774e1721703 [term 2 FOLLOWER]: Refusing update from remote peer 53b1888e4c9546a5af572aa5c87b29e2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260926 11:32:31.553006 15687 consensus_queue.cc:1035] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260926 11:32:31.555298 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:31.554913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.1:48475)
0926 11:32:31.555130 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:31.555194 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":62,"threads_started":1}
I20260926 11:32:31.557741 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:31.557559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.1:46775)
0926 11:32:31.557648 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:31.557689 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260926 11:32:31.557739 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:31.557405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.2:43469)
0926 11:32:31.557591 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:31.557646 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
I20260926 11:32:31.878417 15700 ts_tablet_manager.cc:927] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5: Initiating tablet copy from peer 53b1888e4c9546a5af572aa5c87b29e2 (127.0.98.2:36243)
I20260926 11:32:31.879045 15700 tablet_copy_client.cc:316] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.2:36243
Sep 26 11:32:31 dist-test-slave-mrck krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260926 11:32:31.894558 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:31.879580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36243 (local address 127.0.98.3:55987)
0926 11:32:31.879889 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:31.879907 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:31.880025 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:31.880525 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:31.880530 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:31.880915 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:32:31.881028 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:31.881033 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:31.881983 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:31.881986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:31.882641 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:31.882650 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:31.883338 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:31.883341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:31.883482 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:31.888888 (+ 5406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:32:31.888903 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:32:31.890417 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:32:31.892586 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:31.892597 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:31.892604 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:31.892846 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:31.893166 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:31.893171 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:31.893173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:31.893281 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:31.893792 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:32:31.893800 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:32:31.893937 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 11:32:31.894096 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 11:32:31.894315 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":116,"threads_started":1}
I20260926 11:32:31.894551 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:31.879771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55987 (local address 127.0.98.2:36243)
0926 11:32:31.880128 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:31.880137 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:31.880158 (+ 21us) server_negotiation.cc:407] Connection header received
0926 11:32:31.880204 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:31.880208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:31.880385 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:31.880517 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:31.881137 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:31.881859 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:31.882792 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:31.883216 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:31.883336 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:31.890614 (+ 7278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:31.890626 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:31.890641 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:31.890668 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:31.892432 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:31.892976 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:31.892985 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:31.892991 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:31.893070 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:31.893375 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:31.893379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:31.893381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:31.893661 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:31.893824 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:31.894201 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 11:32:31.894359 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":90,"threads_started":1}
I20260926 11:32:31.895798 15520 tablet_copy_service.cc:140] P 53b1888e4c9546a5af572aa5c87b29e2: Received BeginTabletCopySession request for tablet cbe3811fe91a4ec082cadd93d13cb307 from peer 133399fee1e447aba4e74af5695a6dd5 ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55987)
I20260926 11:32:31.896003 15520 tablet_copy_service.cc:161] P 53b1888e4c9546a5af572aa5c87b29e2: Beginning new tablet copy session on tablet cbe3811fe91a4ec082cadd93d13cb307 from peer 133399fee1e447aba4e74af5695a6dd5 at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55987: session id = 133399fee1e447aba4e74af5695a6dd5-cbe3811fe91a4ec082cadd93d13cb307
I20260926 11:32:31.897156 15520 tablet_copy_source_session.cc:205] T cbe3811fe91a4ec082cadd93d13cb307 P 53b1888e4c9546a5af572aa5c87b29e2: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 11:32:31.901198 15700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbe3811fe91a4ec082cadd93d13cb307. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:32:31.907943 15700 tablet_copy_client.cc:791] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5: tablet copy: Starting download of 0 data blocks...
I20260926 11:32:31.908147 15700 tablet_copy_client.cc:655] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5: tablet copy: Starting download of 1 WAL segments...
I20260926 11:32:31.909973 15700 tablet_copy_client.cc:531] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 11:32:31.911814 15700 tablet_bootstrap.cc:492] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5: Bootstrap starting.
I20260926 11:32:31.917353 15700 log.cc:824] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5: Log is configured to *not* fsync() on all Append() calls
I20260926 11:32:31.923727 15700 tablet_bootstrap.cc:492] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 11:32:31.924086 15700 tablet_bootstrap.cc:492] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5: Bootstrap complete.
I20260926 11:32:31.924389 15700 ts_tablet_manager.cc:1397] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.003s
I20260926 11:32:31.926631 15700 raft_consensus.cc:348] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133399fee1e447aba4e74af5695a6dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37853 } } peers { permanent_uuid: "53b1888e4c9546a5af572aa5c87b29e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36243 } } peers { permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 } }
I20260926 11:32:31.926905 15700 raft_consensus.cc:729] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 133399fee1e447aba4e74af5695a6dd5, State: Initialized, Role: FOLLOWER
I20260926 11:32:31.927372 15700 consensus_queue.cc:260] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "133399fee1e447aba4e74af5695a6dd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37853 } } peers { permanent_uuid: "53b1888e4c9546a5af572aa5c87b29e2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36243 } } peers { permanent_uuid: "b611f7b8e9584eb4928bd774e1721703" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39175 } }
I20260926 11:32:31.928401 15700 ts_tablet_manager.cc:1428] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260926 11:32:31.928570 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:31.928303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.3:60663)
0926 11:32:31.928409 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:31.928479 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 11:32:31.929076 15520 tablet_copy_service.cc:342] P 53b1888e4c9546a5af572aa5c87b29e2: Request end of tablet copy session 133399fee1e447aba4e74af5695a6dd5-cbe3811fe91a4ec082cadd93d13cb307 received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55987
I20260926 11:32:31.929198 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:31.929000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.3:55645)
0926 11:32:31.929088 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:31.929139 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 11:32:31.929198 15520 tablet_copy_service.cc:433] P 53b1888e4c9546a5af572aa5c87b29e2: ending tablet copy session 133399fee1e447aba4e74af5695a6dd5-cbe3811fe91a4ec082cadd93d13cb307 on tablet cbe3811fe91a4ec082cadd93d13cb307 with peer 133399fee1e447aba4e74af5695a6dd5
I20260926 11:32:32.387801 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:32.387382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.2:39823)
0926 11:32:32.387604 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:32.387650 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":64,"threads_started":1}
I20260926 11:32:32.387857 15634 raft_consensus.cc:1203] T cbe3811fe91a4ec082cadd93d13cb307 P 133399fee1e447aba4e74af5695a6dd5 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260926 11:32:32.388805 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:32.388555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.3:58967)
0926 11:32:32.388691 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:32.388736 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 11:32:32.558949 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:32.558529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.1:33159)
0926 11:32:32.558786 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:32.558849 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":67,"threads_started":1}
I20260926 11:32:33.388881 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:33.388500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.2:35611)
0926 11:32:33.388733 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:33.388785 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
I20260926 11:32:33.389770 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:33.389429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.3:39093)
0926 11:32:33.389645 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:33.389695 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":64,"threads_started":1}
I20260926 11:32:33.560304 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:33.559866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.1:55543)
0926 11:32:33.560137 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:33.560202 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":80,"threads_started":1}
I20260926 11:32:34.390084 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:34.389694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.2:41995)
0926 11:32:34.389938 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:34.389991 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":77,"threads_started":1}
I20260926 11:32:34.390846 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:34.390428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.3:44239)
0926 11:32:34.390708 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:34.390758 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":87,"threads_started":1}
I20260926 11:32:34.561537 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:34.561090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.1:56271)
0926 11:32:34.561350 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:34.561437 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":77,"threads_started":1}
I20260926 11:32:35.391682 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:35.391138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.2:48179)
0926 11:32:35.391485 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:35.391573 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":99,"threads_started":1}
I20260926 11:32:35.392272 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:35.391851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.3:44435)
0926 11:32:35.392129 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:35.392196 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":85,"threads_started":1}
I20260926 11:32:35.392477 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:35.392247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.2:51655)
0926 11:32:35.392363 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:35.392414 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 11:32:35.562912 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:35.562461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.1:42731)
0926 11:32:35.562738 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:35.562794 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":79,"threads_started":1}
I20260926 11:32:36.393618 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:36.393124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.3:39541)
0926 11:32:36.393412 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:36.393486 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":88,"threads_started":1}
I20260926 11:32:36.393661 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:36.393197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.2:38403)
0926 11:32:36.393491 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:36.393556 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":107,"threads_started":1}
I20260926 11:32:36.564314 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:36.563873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.1:35077)
0926 11:32:36.564145 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:36.564211 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":88,"threads_started":1}
I20260926 11:32:37.394972 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:37.394495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.3:40679)
0926 11:32:37.394802 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:37.394873 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":113,"threads_started":1}
I20260926 11:32:37.395017 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:37.394556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.2:52123)
0926 11:32:37.394857 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:37.394914 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":59,"threads_started":1}
I20260926 11:32:37.565707 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:37.565188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.1:35779)
0926 11:32:37.565525 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:37.565594 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":97,"threads_started":1}
I20260926 11:32:38.396407 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:38.395921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.2:53203)
0926 11:32:38.396202 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:38.396280 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":100,"threads_started":1}
I20260926 11:32:38.396482 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:38.395935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.3:51563)
0926 11:32:38.396299 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:38.396352 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":89,"threads_started":1}
I20260926 11:32:38.567191 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:38.566689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.1:34983)
0926 11:32:38.566999 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:38.567086 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":92,"threads_started":1}
I20260926 11:32:39.397694 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:39.397259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.2:39619)
0926 11:32:39.397545 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:39.397601 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":102,"threads_started":1}
I20260926 11:32:39.397778 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:39.397324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.3:59451)
0926 11:32:39.397629 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:39.397695 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":112,"threads_started":1}
I20260926 11:32:39.568779 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:39.568217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.1:40805)
0926 11:32:39.568546 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:39.568650 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":128,"threads_started":1}
I20260926 11:32:40.399060 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:40.398639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.3:47213)
0926 11:32:40.398893 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:40.398958 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":100,"threads_started":1}
I20260926 11:32:40.399088 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:40.398657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.2:36179)
0926 11:32:40.398945 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:40.399003 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":119,"threads_started":1}
I20260926 11:32:40.570189 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:40.569737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.1:49273)
0926 11:32:40.570019 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:40.570088 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":78,"threads_started":1}
I20260926 11:32:41.400467 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:41.399942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.3:43883)
0926 11:32:41.400268 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:41.400346 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":71,"threads_started":1}
I20260926 11:32:41.400626 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:41.400094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.2:47665)
0926 11:32:41.400415 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:41.400503 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":81,"threads_started":1}
I20260926 11:32:41.571512 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:41.571098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.1:38135)
0926 11:32:41.571350 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:41.571413 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":76,"threads_started":1}
I20260926 11:32:42.401965 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:42.401500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.3:56409)
0926 11:32:42.401800 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:42.401865 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":91,"threads_started":1}
I20260926 11:32:42.402001 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:42.401500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.2:57753)
0926 11:32:42.401863 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:42.401917 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":96,"threads_started":1}
I20260926 11:32:42.572762 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:42.572351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.1:45461)
0926 11:32:42.572619 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:42.572678 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":99,"threads_started":1}
I20260926 11:32:43.403406 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:43.402919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.2:43051)
0926 11:32:43.403208 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:43.403301 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":86,"threads_started":1}
I20260926 11:32:43.403424 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:43.402919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.3:35531)
0926 11:32:43.403280 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:43.403338 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":96,"threads_started":1}
I20260926 11:32:43.573951 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:43.573517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.1:42883)
0926 11:32:43.573776 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:43.573847 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":72,"threads_started":1}
I20260926 11:32:44.404706 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:44.404284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.2:37395)
0926 11:32:44.404556 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:44.404610 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":77,"threads_started":1}
I20260926 11:32:44.404706 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:44.404330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.3:35117)
0926 11:32:44.404567 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:44.404620 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":85,"threads_started":1}
I20260926 11:32:44.575130 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:44.574756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36827 (local address 127.0.98.1:45881)
0926 11:32:44.574996 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:44.575045 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":91,"threads_started":1}
I20260926 11:32:44.992805 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 15247
I20260926 11:32:44.998684 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 15415
I20260926 11:32:45.004582 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 15549
2026-09-26T11:32:45Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19430 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:32:45 dist-test-slave-mrck krb5kdc[15749](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:32:45 dist-test-slave-mrck krb5kdc[15749](info): set up 2 sockets
Sep 26 11:32:45 dist-test-slave-mrck krb5kdc[15749](info): commencing operation
krb5kdc: starting...
W20260926 11:32:46.068059 392 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:32:46 dist-test-slave-mrck krb5kdc[15749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:32:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:32:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:32:46.190609 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44617
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:32:46.305068 15762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:32:46.305378 15762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:32:46.305423 15762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:32:46.309551 15762 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 11:32:46.309628 15762 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 11:32:46.309643 15762 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 11:32:46.309657 15762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:32:46.309677 15762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:32:46.309695 15762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:32:46.309711 15762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:32:46.309726 15762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:32:46.315333 15762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:32:46.316607 15762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:32:46.317600 15762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:32:46.326892 15767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:46.327075 15768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:46.327162 15762 server_base.cc:1034] running on GCE node
W20260926 11:32:46.327075 15770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:46.327930 15762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:32:46.329159 15762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:32:46.330379 15762 hybrid_clock.cc:648] HybridClock initialized: now 1790422366330334 us; error 43 us; skew 500 ppm
Sep 26 11:32:46 dist-test-slave-mrck krb5kdc[15749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:32:46.333480 15762 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:32:46.334481 15762 webserver.cc:460] Webserver started at http://127.0.98.62:40365/ using document root <none> and password file <none>
I20260926 11:32:46.335096 15762 fs_manager.cc:359] Metadata directory not provided
I20260926 11:32:46.335191 15762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:32:46.335444 15762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:32:46.337716 15762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "4d9b63b3aa6645f49a00ad00664d0c09"
format_stamp: "Formatted at 2026-09-26 11:32:46 on dist-test-slave-mrck"
server_key: "dfe14d6a598c2a4601ddbf7f9c0ab01a"
server_key_iv: "617d94376fa144fc1b605ae895432b15"
server_key_version: "encryptionkey@0"
I20260926 11:32:46.338323 15762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "4d9b63b3aa6645f49a00ad00664d0c09"
format_stamp: "Formatted at 2026-09-26 11:32:46 on dist-test-slave-mrck"
server_key: "dfe14d6a598c2a4601ddbf7f9c0ab01a"
server_key_iv: "617d94376fa144fc1b605ae895432b15"
server_key_version: "encryptionkey@0"
I20260926 11:32:46.342401 15762 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 11:32:46.345203 15777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:46.346712 15762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 11:32:46.346884 15762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "4d9b63b3aa6645f49a00ad00664d0c09"
format_stamp: "Formatted at 2026-09-26 11:32:46 on dist-test-slave-mrck"
server_key: "dfe14d6a598c2a4601ddbf7f9c0ab01a"
server_key_iv: "617d94376fa144fc1b605ae895432b15"
server_key_version: "encryptionkey@0"
I20260926 11:32:46.347045 15762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:32:46.374372 15762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:32:46.378329 15762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:32:46.378575 15762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:32:46.386400 15762 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44617
I20260926 11:32:46.386699 15829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44617 every 8 connection(s)
I20260926 11:32:46.387785 15762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:32:46.390698 15830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:32:46.396414 15830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09: Bootstrap starting.
I20260926 11:32:46.397357 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 15762
I20260926 11:32:46.397503 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:32:46.397796 392 external_mini_cluster.cc:1428] Setting key f5cb674073a6006c2bf79555b6209a30
I20260926 11:32:46.399214 15830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09: Neither blocks nor log segments found. Creating new log.
I20260926 11:32:46.400028 15830 log.cc:824] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09: Log is configured to *not* fsync() on all Append() calls
I20260926 11:32:46.401939 15830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09: No bootstrap required, opened a new log
I20260926 11:32:46.405473 15830 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d9b63b3aa6645f49a00ad00664d0c09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44617 } }
I20260926 11:32:46.405699 15830 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:32:46.405771 15830 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d9b63b3aa6645f49a00ad00664d0c09, State: Initialized, Role: FOLLOWER
I20260926 11:32:46.406301 15830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d9b63b3aa6645f49a00ad00664d0c09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44617 } }
I20260926 11:32:46.406481 15830 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:32:46.406579 15830 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
Sep 26 11:32:46 dist-test-slave-mrck krb5kdc[15749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:32:46.406689 15830 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:32:46.407987 15830 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d9b63b3aa6645f49a00ad00664d0c09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44617 } }
I20260926 11:32:46.408414 15830 leader_election.cc:302] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d9b63b3aa6645f49a00ad00664d0c09; no voters:
I20260926 11:32:46.408802 15830 leader_election.cc:288] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:32:46.408960 15835 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:32:46.409225 15835 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [term 1 LEADER]: Becoming Leader. State: Replica: 4d9b63b3aa6645f49a00ad00664d0c09, State: Running, Role: LEADER
I20260926 11:32:46.409631 15835 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d9b63b3aa6645f49a00ad00664d0c09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44617 } }
I20260926 11:32:46.410218 15830 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:32:46.411674 15837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d9b63b3aa6645f49a00ad00664d0c09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d9b63b3aa6645f49a00ad00664d0c09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44617 } } }
I20260926 11:32:46.411827 15837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [sys.catalog]: This master's current role is: LEADER
I20260926 11:32:46.412204 15844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:32:46.412631 15836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d9b63b3aa6645f49a00ad00664d0c09. Latest consensus state: current_term: 1 leader_uuid: "4d9b63b3aa6645f49a00ad00664d0c09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d9b63b3aa6645f49a00ad00664d0c09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44617 } } }
I20260926 11:32:46.412719 15836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [sys.catalog]: This master's current role is: LEADER
I20260926 11:32:46.413163 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:46.398937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46034 (local address 127.0.98.62:44617)
0926 11:32:46.399458 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:46.399474 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:46.399509 (+ 35us) server_negotiation.cc:407] Connection header received
0926 11:32:46.400301 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:46.400321 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:46.400644 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:46.400984 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:46.401867 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:46.402907 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:46.403794 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:46.404307 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:46.404493 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:46.407299 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:46.407306 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:46.407324 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:46.407357 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:46.409434 (+ 2077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:46.409984 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:46.409990 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:46.409996 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:46.410138 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:46.410495 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:46.410498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:46.410500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:46.411046 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:46.411254 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:46.411689 (+ 435us) server_negotiation.cc:299] Negotiation successful
0926 11:32:46.412055 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":148,"threads_started":1}
I20260926 11:32:46.415550 15844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:32:46.421679 15844 catalog_manager.cc:1347] Generated new cluster ID: 065689bab73e4a5db2358bc164e61a5c
I20260926 11:32:46.421759 15844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:32:46.427448 15844 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:32:46.428730 15844 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:32:46.440904 15844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09: Generated new TSK 0
I20260926 11:32:46.441810 15844 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:32:46.470816 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44617
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:32:46.583485 15856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:32:46.583840 15856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:32:46.583923 15856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:32:46.588037 15856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:32:46.588099 15856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:32:46.588181 15856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:32:46.593590 15856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:32:46.594745 15856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:32:46.595829 15856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:32:46.605818 15864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:46.605872 15861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:46.606026 15862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:46.606307 15856 server_base.cc:1034] running on GCE node
I20260926 11:32:46.606791 15856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:32:46.607561 15856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:32:46.608779 15856 hybrid_clock.cc:648] HybridClock initialized: now 1790422366608752 us; error 34 us; skew 500 ppm
Sep 26 11:32:46 dist-test-slave-mrck krb5kdc[15749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:32:46.612006 15856 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:32:46.613106 15856 webserver.cc:460] Webserver started at http://127.0.98.1:39597/ using document root <none> and password file <none>
I20260926 11:32:46.613705 15856 fs_manager.cc:359] Metadata directory not provided
I20260926 11:32:46.613799 15856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:32:46.614046 15856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:32:46.616263 15856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/data/instance:
uuid: "119a39a719be45f88987bfa5a856ace8"
format_stamp: "Formatted at 2026-09-26 11:32:46 on dist-test-slave-mrck"
server_key: "b0201dab8a58429f825ebe42dfa32cb1"
server_key_iv: "c091791fcf35001a00da506b2e31b5a3"
server_key_version: "encryptionkey@0"
I20260926 11:32:46.616827 15856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "119a39a719be45f88987bfa5a856ace8"
format_stamp: "Formatted at 2026-09-26 11:32:46 on dist-test-slave-mrck"
server_key: "b0201dab8a58429f825ebe42dfa32cb1"
server_key_iv: "c091791fcf35001a00da506b2e31b5a3"
server_key_version: "encryptionkey@0"
I20260926 11:32:46.620671 15856 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 11:32:46.623211 15871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:46.624512 15856 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:32:46.624652 15856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "119a39a719be45f88987bfa5a856ace8"
format_stamp: "Formatted at 2026-09-26 11:32:46 on dist-test-slave-mrck"
server_key: "b0201dab8a58429f825ebe42dfa32cb1"
server_key_iv: "c091791fcf35001a00da506b2e31b5a3"
server_key_version: "encryptionkey@0"
I20260926 11:32:46.624778 15856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:32:46.656906 15856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:32:46.660203 15856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:32:46.660421 15856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:32:46.661036 15856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:32:46.661972 15856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:32:46.662043 15856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:46.662117 15856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:32:46.662166 15856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:46.672497 15856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43321
I20260926 11:32:46.672528 15984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43321 every 8 connection(s)
I20260926 11:32:46.673449 15856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:32:46.678181 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 15856
I20260926 11:32:46.678336 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:32:46.678699 392 external_mini_cluster.cc:1428] Setting key 9a0a3781a07268b5a8749468f589069b
Sep 26 11:32:46 dist-test-slave-mrck krb5kdc[15749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 11:32:46.690053 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:46.675449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55029 (local address 127.0.98.62:44617)
0926 11:32:46.675631 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:46.675636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:46.676811 (+ 1175us) server_negotiation.cc:407] Connection header received
0926 11:32:46.677750 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:46.677754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:46.677803 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:46.677902 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:46.679570 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:46.680365 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:46.682398 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:46.682808 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:46.682932 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:46.685867 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:46.685875 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:46.685881 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:46.685919 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:46.687705 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:46.688261 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:46.688265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:46.688267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:46.688362 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:46.688697 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:46.688703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:46.688708 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:46.688979 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:46.689086 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:46.689796 (+ 710us) server_negotiation.cc:299] Negotiation successful
0926 11:32:46.689926 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 11:32:46.690975 15987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:46.675938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.1:55029)
0926 11:32:46.676651 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:46.676696 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:46.677535 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:46.678154 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:46.678166 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:46.678788 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:32:46.679430 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:46.679446 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:46.680504 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:46.680508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:46.682206 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:46.682223 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:46.682983 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:46.682986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:46.683140 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:46.683873 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:32:46.683890 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:32:46.685661 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:32:46.687872 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:46.687877 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:46.687890 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:46.688155 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:46.688463 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:46.688465 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:46.688467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:46.688598 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:46.689155 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:32:46.689161 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:32:46.689546 (+ 385us) client_negotiation.cc:769] Sending connection context
0926 11:32:46.689731 (+ 185us) client_negotiation.cc:240] Negotiation successful
0926 11:32:46.690024 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":109,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 11:32:46.692513 15985 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44617
I20260926 11:32:46.692793 15985 heartbeater.cc:464] Registering TS with master...
I20260926 11:32:46.693495 15985 heartbeater.cc:511] Master 127.0.98.62:44617 requested a full tablet report, sending...
I20260926 11:32:46.695173 15794 ts_manager.cc:194] Registered new tserver with Master: 119a39a719be45f88987bfa5a856ace8 (127.0.98.1:43321)
I20260926 11:32:46.696444 15794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55029
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:32:46.707585 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44617
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:32:46.822120 15990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:32:46.822448 15990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:32:46.822525 15990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:32:46.826629 15990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:32:46.826746 15990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:32:46.826860 15990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:32:46.832403 15990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:32:46.833487 15990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:32:46.834522 15990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:32:46.844442 15998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:46.844437 15995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:46.844442 15996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:46.844831 15990 server_base.cc:1034] running on GCE node
I20260926 11:32:46.845312 15990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:32:46.846019 15990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:32:46.847175 15990 hybrid_clock.cc:648] HybridClock initialized: now 1790422366847170 us; error 38 us; skew 500 ppm
Sep 26 11:32:46 dist-test-slave-mrck krb5kdc[15749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:32:46.850060 15990 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:32:46.851064 15990 webserver.cc:460] Webserver started at http://127.0.98.2:46795/ using document root <none> and password file <none>
I20260926 11:32:46.851691 15990 fs_manager.cc:359] Metadata directory not provided
I20260926 11:32:46.851801 15990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:32:46.852033 15990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:32:46.854107 15990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/data/instance:
uuid: "9efbdbfb4d664f32a69fe60963885f21"
format_stamp: "Formatted at 2026-09-26 11:32:46 on dist-test-slave-mrck"
server_key: "a7085119b68a3212b78d9c5abdda1472"
server_key_iv: "dd5f76d2b089eb8e2225aaf35cda8115"
server_key_version: "encryptionkey@0"
I20260926 11:32:46.854682 15990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9efbdbfb4d664f32a69fe60963885f21"
format_stamp: "Formatted at 2026-09-26 11:32:46 on dist-test-slave-mrck"
server_key: "a7085119b68a3212b78d9c5abdda1472"
server_key_iv: "dd5f76d2b089eb8e2225aaf35cda8115"
server_key_version: "encryptionkey@0"
I20260926 11:32:46.858598 15990 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260926 11:32:46.861291 16005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:46.862639 15990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 11:32:46.862811 15990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "9efbdbfb4d664f32a69fe60963885f21"
format_stamp: "Formatted at 2026-09-26 11:32:46 on dist-test-slave-mrck"
server_key: "a7085119b68a3212b78d9c5abdda1472"
server_key_iv: "dd5f76d2b089eb8e2225aaf35cda8115"
server_key_version: "encryptionkey@0"
I20260926 11:32:46.862936 15990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:32:46.880934 15990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:32:46.884473 15990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:32:46.884694 15990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:32:46.885347 15990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:32:46.886221 15990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:32:46.886291 15990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:46.886385 15990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:32:46.886435 15990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:46.895787 15990 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43335
I20260926 11:32:46.895797 16118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43335 every 8 connection(s)
I20260926 11:32:46.896731 15990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:32:46.904354 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 15990
I20260926 11:32:46.904500 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:32:46.904793 392 external_mini_cluster.cc:1428] Setting key 8d227b339ca018389da7b67097f03e58
Sep 26 11:32:46 dist-test-slave-mrck krb5kdc[15749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:32:46.911387 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:46.898603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60351 (local address 127.0.98.62:44617)
0926 11:32:46.898782 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:46.898787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:46.899896 (+ 1109us) server_negotiation.cc:407] Connection header received
0926 11:32:46.900786 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:46.900789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:46.900869 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:46.901000 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:46.902196 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:46.902730 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:46.903946 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:46.904175 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:46.904291 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:46.907186 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:46.907189 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:46.907191 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:46.907216 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:46.908821 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:46.909456 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:46.909463 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:46.909467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:46.909571 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:46.909968 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:46.909974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:46.909979 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:46.910255 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:46.910368 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:46.910936 (+ 568us) server_negotiation.cc:299] Negotiation successful
0926 11:32:46.911133 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 11:32:46.912679 16121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:46.899012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.2:60351)
0926 11:32:46.899682 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:46.899723 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:46.900576 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:46.901177 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:46.901187 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:46.901558 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:32:46.902046 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:46.902059 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:46.902862 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:46.902868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:46.903825 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:46.903834 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:46.904332 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:46.904336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:46.904555 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:46.905268 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:32:46.905286 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:32:46.907002 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:32:46.908965 (+ 1963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:46.908975 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:46.908997 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:46.909333 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:46.909679 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:46.909682 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:46.909683 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:46.909877 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:46.910410 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:32:46.910416 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:32:46.910674 (+ 258us) client_negotiation.cc:769] Sending connection context
0926 11:32:46.910856 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 11:32:46.911311 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":113,"threads_started":1}
I20260926 11:32:46.913915 16119 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44617
I20260926 11:32:46.914187 16119 heartbeater.cc:464] Registering TS with master...
I20260926 11:32:46.914690 16119 heartbeater.cc:511] Master 127.0.98.62:44617 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 11:32:46.915856 15794 ts_manager.cc:194] Registered new tserver with Master: 9efbdbfb4d664f32a69fe60963885f21 (127.0.98.2:43335)
I20260926 11:32:46.916528 15794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60351
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:32:46.934296 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44617
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:32:47.046007 16124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:32:47.046284 16124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:32:47.046330 16124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:32:47.050534 16124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:32:47.050601 16124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:32:47.050662 16124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:32:47.056301 16124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:32:47.057358 16124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:32:47.058342 16124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:32:47.068091 16132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:47.068091 16130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:32:47.068123 16129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:32:47.068459 16124 server_base.cc:1034] running on GCE node
I20260926 11:32:47.068866 16124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:32:47.069553 16124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:32:47.070749 16124 hybrid_clock.cc:648] HybridClock initialized: now 1790422367070727 us; error 34 us; skew 500 ppm
Sep 26 11:32:47 dist-test-slave-mrck krb5kdc[15749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:32:47.073688 16124 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:32:47.074642 16124 webserver.cc:460] Webserver started at http://127.0.98.3:36973/ using document root <none> and password file <none>
I20260926 11:32:47.075179 16124 fs_manager.cc:359] Metadata directory not provided
I20260926 11:32:47.075240 16124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:32:47.075403 16124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:32:47.077533 16124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/data/instance:
uuid: "734e79c295e541f1911a62a82e8c42c8"
format_stamp: "Formatted at 2026-09-26 11:32:47 on dist-test-slave-mrck"
server_key: "c1e319c1370928ee0f125e9a8317fef0"
server_key_iv: "c0f9b0b6715c8979445be1ae5b83570e"
server_key_version: "encryptionkey@0"
I20260926 11:32:47.078045 16124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "734e79c295e541f1911a62a82e8c42c8"
format_stamp: "Formatted at 2026-09-26 11:32:47 on dist-test-slave-mrck"
server_key: "c1e319c1370928ee0f125e9a8317fef0"
server_key_iv: "c0f9b0b6715c8979445be1ae5b83570e"
server_key_version: "encryptionkey@0"
I20260926 11:32:47.081868 16124 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 11:32:47.084319 16139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:47.085660 16124 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:32:47.085776 16124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "734e79c295e541f1911a62a82e8c42c8"
format_stamp: "Formatted at 2026-09-26 11:32:47 on dist-test-slave-mrck"
server_key: "c1e319c1370928ee0f125e9a8317fef0"
server_key_iv: "c0f9b0b6715c8979445be1ae5b83570e"
server_key_version: "encryptionkey@0"
I20260926 11:32:47.085876 16124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:32:47.117406 16124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:32:47.120353 16124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:32:47.120509 16124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:32:47.121035 16124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:32:47.121798 16124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:32:47.121845 16124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:47.121883 16124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:32:47.121907 16124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:32:47.131337 16124 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42483
I20260926 11:32:47.131354 16252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42483 every 8 connection(s)
I20260926 11:32:47.132357 16124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:32:47.141346 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 16124
I20260926 11:32:47.141505 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:32:47.141813 392 external_mini_cluster.cc:1428] Setting key ebc933eb1d2302c4253874b0a93dd4da
Sep 26 11:32:47 dist-test-slave-mrck krb5kdc[15749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:32:47.146332 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:47.134177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44957 (local address 127.0.98.62:44617)
0926 11:32:47.134332 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:47.134336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:47.135150 (+ 814us) server_negotiation.cc:407] Connection header received
0926 11:32:47.136625 (+ 1475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:47.136628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:47.136689 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:47.136769 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:47.137977 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:47.138516 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:47.139687 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:47.139975 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:47.140097 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:47.142814 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:47.142819 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:47.142821 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:47.142853 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:47.144555 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:47.145027 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:47.145029 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:47.145030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:47.145105 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:47.145412 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:47.145414 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:47.145415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:47.145585 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:47.145657 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:47.146111 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 11:32:47.146214 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 11:32:47.147203 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:47.134521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.3:44957)
0926 11:32:47.134992 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:47.135036 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:47.136409 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:47.136944 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:47.136953 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:47.137332 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:32:47.137825 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:47.137838 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:47.138648 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:47.138654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:47.139556 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:47.139565 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:47.140144 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:47.140148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:47.140331 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:47.140886 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:32:47.140901 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:32:47.142603 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:32:47.144656 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:47.144660 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:47.144673 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:47.144941 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:47.145180 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:32:47.145182 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:32:47.145184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:32:47.145336 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:32:47.145666 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:32:47.145671 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:32:47.145909 (+ 238us) client_negotiation.cc:769] Sending connection context
0926 11:32:47.146066 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 11:32:47.146346 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":117,"threads_started":1}
I20260926 11:32:47.148468 16253 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44617
I20260926 11:32:47.148722 16253 heartbeater.cc:464] Registering TS with master...
I20260926 11:32:47.149222 16253 heartbeater.cc:511] Master 127.0.98.62:44617 requested a full tablet report, sending...
I20260926 11:32:47.150386 15794 ts_manager.cc:194] Registered new tserver with Master: 734e79c295e541f1911a62a82e8c42c8 (127.0.98.3:42483)
I20260926 11:32:47.150960 15794 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44957
I20260926 11:32:47.156335 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:32:47.165498 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:47.158331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46038 (local address 127.0.98.62:44617)
0926 11:32:47.158488 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:47.158492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:47.158567 (+ 75us) server_negotiation.cc:407] Connection header received
0926 11:32:47.158668 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:47.158670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:47.158708 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:47.158775 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:32:47.159461 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:47.160021 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:47.160837 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:47.161050 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:47.161129 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:47.162522 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:32:47.162525 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:32:47.162527 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:32:47.162547 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:32:47.163868 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:47.164263 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:47.164266 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:47.164267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:47.164374 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:32:47.164622 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:32:47.164625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:32:47.164626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:32:47.164848 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:32:47.164954 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:47.165170 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 11:32:47.165300 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 11:32:47.168598 15794 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:32:47.171622 15794 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:32:47.186664 16268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:47.181132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42483 (local address 127.0.0.1:55822)
0926 11:32:47.181826 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:47.181840 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:47.181950 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:47.182831 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:47.182834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:47.182853 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:32:47.182932 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:47.182937 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:47.184258 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:47.184262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:47.185324 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:47.185333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:47.186117 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:47.186120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:47.186237 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:47.186275 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 11:32:47.186390 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 11:32:47.186512 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":62,"thread_start_us":94,"threads_started":1}
I20260926 11:32:47.186674 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:47.181551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55822 (local address 127.0.98.3:42483)
0926 11:32:47.182413 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:47.182427 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:47.182449 (+ 22us) server_negotiation.cc:407] Connection header received
0926 11:32:47.182520 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:47.182524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:47.182689 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:47.182834 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:32:47.183045 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:47.184101 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:47.185447 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:47.185989 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:47.186116 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:47.186284 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:47.186427 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 11:32:47.186517 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":753,"thread_start_us":142,"threads_started":1}
I20260926 11:32:47.187047 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:47.180547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43335 (local address 127.0.0.1:47982)
0926 11:32:47.181471 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:47.181485 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:47.181581 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:47.182140 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:47.182147 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:47.182185 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:32:47.182356 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:47.182368 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:47.183528 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:47.183534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:47.185186 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:47.185195 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:47.186495 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:47.186498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:47.186711 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:47.186752 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 11:32:47.186818 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 11:32:47.186893 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"mutex_wait_us":211,"spinlock_wait_cycles":7296,"thread_start_us":87,"threads_started":1}
I20260926 11:32:47.187152 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:47.180500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42300 (local address 127.0.98.1:43321)
0926 11:32:47.181303 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:47.181318 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:47.181343 (+ 25us) server_negotiation.cc:407] Connection header received
0926 11:32:47.181588 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:47.181595 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:47.181825 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:47.181989 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:32:47.182500 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:47.183689 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:47.185312 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:47.186174 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:47.186288 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:47.186430 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:47.186891 (+ 461us) server_negotiation.cc:299] Negotiation successful
0926 11:32:47.187000 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":78,"threads_started":1}
I20260926 11:32:47.187623 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:47.180796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47982 (local address 127.0.98.2:43335)
0926 11:32:47.181303 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:47.181317 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:47.181685 (+ 368us) server_negotiation.cc:407] Connection header received
0926 11:32:47.181771 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:47.181776 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:47.181977 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:47.182146 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:32:47.182478 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:47.183413 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:47.185482 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:47.186214 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:47.186303 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:47.186430 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:47.187353 (+ 923us) server_negotiation.cc:299] Negotiation successful
0926 11:32:47.187478 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":145,"threads_started":1}
I20260926 11:32:47.187767 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:47.180343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43321 (local address 127.0.0.1:42300)
0926 11:32:47.181173 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:47.181232 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:47.181376 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:47.182150 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:47.182152 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:47.182184 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:32:47.182361 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:47.182368 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:47.183858 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:47.183861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:47.185185 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:47.185195 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:47.186495 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:47.186498 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:47.186705 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:47.186744 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 11:32:47.187539 (+ 795us) client_negotiation.cc:240] Negotiation successful
0926 11:32:47.187609 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":216,"spinlock_wait_cycles":3200,"thread_start_us":119,"threads_started":1}
I20260926 11:32:47.190267 16053 tablet_service.cc:1467] Processing CreateTablet for tablet aa54d26157ab43468b2ae3ddcc3ddf66 (DEFAULT_TABLE table=test-table [id=178835abd1464b4d8a4f6271b51f96a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:32:47.190304 16187 tablet_service.cc:1467] Processing CreateTablet for tablet aa54d26157ab43468b2ae3ddcc3ddf66 (DEFAULT_TABLE table=test-table [id=178835abd1464b4d8a4f6271b51f96a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:32:47.190404 15919 tablet_service.cc:1467] Processing CreateTablet for tablet aa54d26157ab43468b2ae3ddcc3ddf66 (DEFAULT_TABLE table=test-table [id=178835abd1464b4d8a4f6271b51f96a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:32:47.191337 16053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa54d26157ab43468b2ae3ddcc3ddf66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:32:47.191371 15919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa54d26157ab43468b2ae3ddcc3ddf66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:32:47.191368 16187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa54d26157ab43468b2ae3ddcc3ddf66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:32:47.197679 16270 tablet_bootstrap.cc:492] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21: Bootstrap starting.
I20260926 11:32:47.199030 16272 tablet_bootstrap.cc:492] T aa54d26157ab43468b2ae3ddcc3ddf66 P 119a39a719be45f88987bfa5a856ace8: Bootstrap starting.
I20260926 11:32:47.200112 16271 tablet_bootstrap.cc:492] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8: Bootstrap starting.
I20260926 11:32:47.200465 16270 tablet_bootstrap.cc:654] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21: Neither blocks nor log segments found. Creating new log.
I20260926 11:32:47.201093 16272 tablet_bootstrap.cc:654] T aa54d26157ab43468b2ae3ddcc3ddf66 P 119a39a719be45f88987bfa5a856ace8: Neither blocks nor log segments found. Creating new log.
I20260926 11:32:47.201231 16270 log.cc:824] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21: Log is configured to *not* fsync() on all Append() calls
I20260926 11:32:47.201881 16272 log.cc:824] T aa54d26157ab43468b2ae3ddcc3ddf66 P 119a39a719be45f88987bfa5a856ace8: Log is configured to *not* fsync() on all Append() calls
I20260926 11:32:47.202659 16271 tablet_bootstrap.cc:654] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8: Neither blocks nor log segments found. Creating new log.
I20260926 11:32:47.203475 16271 log.cc:824] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8: Log is configured to *not* fsync() on all Append() calls
I20260926 11:32:47.203675 16270 tablet_bootstrap.cc:492] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21: No bootstrap required, opened a new log
I20260926 11:32:47.203927 16270 ts_tablet_manager.cc:1397] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260926 11:32:47.204598 16272 tablet_bootstrap.cc:492] T aa54d26157ab43468b2ae3ddcc3ddf66 P 119a39a719be45f88987bfa5a856ace8: No bootstrap required, opened a new log
I20260926 11:32:47.204782 16272 ts_tablet_manager.cc:1397] T aa54d26157ab43468b2ae3ddcc3ddf66 P 119a39a719be45f88987bfa5a856ace8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 11:32:47.205443 16271 tablet_bootstrap.cc:492] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8: No bootstrap required, opened a new log
I20260926 11:32:47.205614 16271 ts_tablet_manager.cc:1397] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260926 11:32:47.207161 16270 raft_consensus.cc:348] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a39a719be45f88987bfa5a856ace8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43321 } } peers { permanent_uuid: "734e79c295e541f1911a62a82e8c42c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42483 } } peers { permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 } }
I20260926 11:32:47.207486 16270 raft_consensus.cc:374] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:32:47.207540 16270 raft_consensus.cc:729] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9efbdbfb4d664f32a69fe60963885f21, State: Initialized, Role: FOLLOWER
I20260926 11:32:47.208019 16270 consensus_queue.cc:260] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a39a719be45f88987bfa5a856ace8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43321 } } peers { permanent_uuid: "734e79c295e541f1911a62a82e8c42c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42483 } } peers { permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 } }
I20260926 11:32:47.208699 16119 heartbeater.cc:503] Master 127.0.98.62:44617 was elected leader, sending a full tablet report...
I20260926 11:32:47.209072 16270 ts_tablet_manager.cc:1428] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 11:32:47.209409 16271 raft_consensus.cc:348] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a39a719be45f88987bfa5a856ace8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43321 } } peers { permanent_uuid: "734e79c295e541f1911a62a82e8c42c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42483 } } peers { permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 } }
I20260926 11:32:47.209679 16271 raft_consensus.cc:374] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:32:47.209725 16271 raft_consensus.cc:729] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 734e79c295e541f1911a62a82e8c42c8, State: Initialized, Role: FOLLOWER
I20260926 11:32:47.209610 16272 raft_consensus.cc:348] T aa54d26157ab43468b2ae3ddcc3ddf66 P 119a39a719be45f88987bfa5a856ace8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a39a719be45f88987bfa5a856ace8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43321 } } peers { permanent_uuid: "734e79c295e541f1911a62a82e8c42c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42483 } } peers { permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 } }
I20260926 11:32:47.209834 16272 raft_consensus.cc:374] T aa54d26157ab43468b2ae3ddcc3ddf66 P 119a39a719be45f88987bfa5a856ace8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:32:47.209875 16272 raft_consensus.cc:729] T aa54d26157ab43468b2ae3ddcc3ddf66 P 119a39a719be45f88987bfa5a856ace8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 119a39a719be45f88987bfa5a856ace8, State: Initialized, Role: FOLLOWER
I20260926 11:32:47.210245 16271 consensus_queue.cc:260] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a39a719be45f88987bfa5a856ace8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43321 } } peers { permanent_uuid: "734e79c295e541f1911a62a82e8c42c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42483 } } peers { permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 } }
I20260926 11:32:47.210244 16272 consensus_queue.cc:260] T aa54d26157ab43468b2ae3ddcc3ddf66 P 119a39a719be45f88987bfa5a856ace8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a39a719be45f88987bfa5a856ace8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43321 } } peers { permanent_uuid: "734e79c295e541f1911a62a82e8c42c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42483 } } peers { permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 } }
I20260926 11:32:47.211128 15985 heartbeater.cc:503] Master 127.0.98.62:44617 was elected leader, sending a full tablet report...
I20260926 11:32:47.211306 16271 ts_tablet_manager.cc:1428] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260926 11:32:47.211306 16272 ts_tablet_manager.cc:1428] T aa54d26157ab43468b2ae3ddcc3ddf66 P 119a39a719be45f88987bfa5a856ace8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 11:32:47.211681 16253 heartbeater.cc:503] Master 127.0.98.62:44617 was elected leader, sending a full tablet report...
W20260926 11:32:47.383345 16254 tablet.cc:2369] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:32:47.390661 16277 raft_consensus.cc:482] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:32:47.390877 16277 raft_consensus.cc:504] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a39a719be45f88987bfa5a856ace8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43321 } } peers { permanent_uuid: "734e79c295e541f1911a62a82e8c42c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42483 } } peers { permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 } }
I20260926 11:32:47.392110 16277 leader_election.cc:288] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 119a39a719be45f88987bfa5a856ace8 (127.0.98.1:43321), 9efbdbfb4d664f32a69fe60963885f21 (127.0.98.2:43335)
I20260926 11:32:47.396497 16267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:47.393140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60813 (local address 127.0.98.2:43335)
0926 11:32:47.393321 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:47.393325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:47.393339 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:32:47.393386 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:47.393389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:47.393435 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:47.393519 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:32:47.393801 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:47.394383 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:47.395412 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:47.395872 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:47.395987 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:47.396048 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:47.396268 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 11:32:47.396343 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 11:32:47.396641 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:47.392333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43321 (local address 127.0.98.3:58973)
0926 11:32:47.392522 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:47.392540 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:47.392638 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:47.393008 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:47.393010 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:47.393025 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:32:47.393122 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:47.393126 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:47.394094 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:47.394097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:47.395345 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:47.395359 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:47.396216 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:47.396221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:47.396343 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:47.396365 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 11:32:47.396442 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 11:32:47.396510 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":1}
I20260926 11:32:47.396664 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:47.392430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58973 (local address 127.0.98.1:43321)
0926 11:32:47.392776 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:47.392781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:47.392796 (+ 15us) server_negotiation.cc:407] Connection header received
0926 11:32:47.392851 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:47.392854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:47.392902 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:47.392981 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:32:47.393285 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:47.393983 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:47.395489 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:47.396094 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:47.396197 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:47.396257 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:47.396469 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 11:32:47.396545 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245}
I20260926 11:32:47.396869 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:47.392365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43335 (local address 127.0.98.3:60813)
0926 11:32:47.392656 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:47.392670 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:32:47.392749 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:32:47.393551 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:32:47.393554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:32:47.393572 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:32:47.393683 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:47.393689 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:47.394503 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:47.394507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:47.395287 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:32:47.395297 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:32:47.395993 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:32:47.395997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:32:47.396097 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:47.396125 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 11:32:47.396630 (+ 505us) client_negotiation.cc:240] Negotiation successful
0926 11:32:47.396707 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":105,"threads_started":1}
I20260926 11:32:47.397599 16073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa54d26157ab43468b2ae3ddcc3ddf66" candidate_uuid: "734e79c295e541f1911a62a82e8c42c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efbdbfb4d664f32a69fe60963885f21" is_pre_election: true
I20260926 11:32:47.397595 15939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa54d26157ab43468b2ae3ddcc3ddf66" candidate_uuid: "734e79c295e541f1911a62a82e8c42c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "119a39a719be45f88987bfa5a856ace8" is_pre_election: true
W20260926 11:32:47.397868 16120 tablet.cc:2369] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:32:47.397917 16073 raft_consensus.cc:2454] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 734e79c295e541f1911a62a82e8c42c8 in term 0.
I20260926 11:32:47.397917 15939 raft_consensus.cc:2454] T aa54d26157ab43468b2ae3ddcc3ddf66 P 119a39a719be45f88987bfa5a856ace8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 734e79c295e541f1911a62a82e8c42c8 in term 0.
I20260926 11:32:47.398624 16141 leader_election.cc:302] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 119a39a719be45f88987bfa5a856ace8, 734e79c295e541f1911a62a82e8c42c8, 9efbdbfb4d664f32a69fe60963885f21; no voters:
I20260926 11:32:47.398976 16277 raft_consensus.cc:2781] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:32:47.399065 16277 raft_consensus.cc:482] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:32:47.399114 16277 raft_consensus.cc:3037] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:32:47.400233 16277 raft_consensus.cc:504] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a39a719be45f88987bfa5a856ace8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43321 } } peers { permanent_uuid: "734e79c295e541f1911a62a82e8c42c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42483 } } peers { permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 } }
I20260926 11:32:47.400643 16277 leader_election.cc:288] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8 [CANDIDATE]: Term 1 election: Requested vote from peers 119a39a719be45f88987bfa5a856ace8 (127.0.98.1:43321), 9efbdbfb4d664f32a69fe60963885f21 (127.0.98.2:43335)
I20260926 11:32:47.401023 15939 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa54d26157ab43468b2ae3ddcc3ddf66" candidate_uuid: "734e79c295e541f1911a62a82e8c42c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "119a39a719be45f88987bfa5a856ace8"
I20260926 11:32:47.401139 15939 raft_consensus.cc:3037] T aa54d26157ab43468b2ae3ddcc3ddf66 P 119a39a719be45f88987bfa5a856ace8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:32:47.402110 15939 raft_consensus.cc:2454] T aa54d26157ab43468b2ae3ddcc3ddf66 P 119a39a719be45f88987bfa5a856ace8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 734e79c295e541f1911a62a82e8c42c8 in term 1.
I20260926 11:32:47.402359 16073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa54d26157ab43468b2ae3ddcc3ddf66" candidate_uuid: "734e79c295e541f1911a62a82e8c42c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efbdbfb4d664f32a69fe60963885f21"
I20260926 11:32:47.402428 16142 leader_election.cc:302] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 119a39a719be45f88987bfa5a856ace8, 734e79c295e541f1911a62a82e8c42c8; no voters:
I20260926 11:32:47.402498 16073 raft_consensus.cc:3037] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:32:47.402649 16277 raft_consensus.cc:2781] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:32:47.402933 16277 raft_consensus.cc:686] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8 [term 1 LEADER]: Becoming Leader. State: Replica: 734e79c295e541f1911a62a82e8c42c8, State: Running, Role: LEADER
I20260926 11:32:47.403254 16277 consensus_queue.cc:237] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a39a719be45f88987bfa5a856ace8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43321 } } peers { permanent_uuid: "734e79c295e541f1911a62a82e8c42c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42483 } } peers { permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 } }
I20260926 11:32:47.403657 16073 raft_consensus.cc:2454] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 734e79c295e541f1911a62a82e8c42c8 in term 1.
I20260926 11:32:47.406183 15792 catalog_manager.cc:5495] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 734e79c295e541f1911a62a82e8c42c8 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "734e79c295e541f1911a62a82e8c42c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a39a719be45f88987bfa5a856ace8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "734e79c295e541f1911a62a82e8c42c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 } health_report { overall_health: UNKNOWN } } }
I20260926 11:32:47.421707 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:47.418237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55828 (local address 127.0.98.3:42483)
0926 11:32:47.418386 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 11:32:47.418389 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:32:47.418403 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:32:47.418508 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:32:47.418510 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:32:47.418561 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:32:47.418650 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:32:47.418847 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:47.419324 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:47.420196 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:32:47.420462 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:32:47.420564 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:32:47.420873 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:32:47.421310 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:32:47.421433 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 11:32:47.421519 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 11:32:47.421574 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 11:32:47.424844 15986 tablet.cc:2369] T aa54d26157ab43468b2ae3ddcc3ddf66 P 119a39a719be45f88987bfa5a856ace8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:32:47.428098 16073 raft_consensus.cc:1261] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21 [term 1 FOLLOWER]: Refusing update from remote peer 734e79c295e541f1911a62a82e8c42c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:32:47.428099 15939 raft_consensus.cc:1261] T aa54d26157ab43468b2ae3ddcc3ddf66 P 119a39a719be45f88987bfa5a856ace8 [term 1 FOLLOWER]: Refusing update from remote peer 734e79c295e541f1911a62a82e8c42c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:32:47.428833 16277 consensus_queue.cc:1035] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:32:47.429028 16280 consensus_queue.cc:1035] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "119a39a719be45f88987bfa5a856ace8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:32:47.437273 16284 mvcc.cc:205] Tried to move back new op lower bound from 7333570016977543168 to 7333570016886513664. Current Snapshot: MvccSnapshot[applied={T|T < 7333570016977543168}]
I20260926 11:32:47.445107 15794 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:32:47.445299 15794 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:32:47.447899 15794 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09: Sending DeleteTablet for 3 replicas of tablet aa54d26157ab43468b2ae3ddcc3ddf66
I20260926 11:32:47.448736 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 15856
I20260926 11:32:47.448715 16053 tablet_service.cc:1514] Processing DeleteTablet for tablet aa54d26157ab43468b2ae3ddcc3ddf66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:32:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47982
I20260926 11:32:47.448767 15919 tablet_service.cc:1514] Processing DeleteTablet for tablet aa54d26157ab43468b2ae3ddcc3ddf66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:32:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42300
I20260926 11:32:47.448899 16187 tablet_service.cc:1514] Processing DeleteTablet for tablet aa54d26157ab43468b2ae3ddcc3ddf66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:32:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55822
I20260926 11:32:47.451894 16293 tablet_replica.cc:330] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8: stopping tablet replica
I20260926 11:32:47.452257 16293 raft_consensus.cc:2229] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:32:47.452723 16293 raft_consensus.cc:2258] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:32:47.453111 16292 tablet_replica.cc:330] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21: stopping tablet replica
I20260926 11:32:47.453430 16292 raft_consensus.cc:2229] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:32:47.453737 16292 raft_consensus.cc:2258] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:32:47.454734 16293 ts_tablet_manager.cc:1905] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:32:47.455454 16292 ts_tablet_manager.cc:1905] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:32:47.455997 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 15990
W20260926 11:32:47.456010 15780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43321 (error 108)
I20260926 11:32:47.458966 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:47.458664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43321 (local address 127.0.0.1:42310)
0926 11:32:47.458805 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:47.458876 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 11:32:47.459200 15780 catalog_manager.cc:4584] TS 119a39a719be45f88987bfa5a856ace8 (127.0.98.1:43321): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aa54d26157ab43468b2ae3ddcc3ddf66: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43321: connect: Connection refused (error 111)
I20260926 11:32:47.460315 16293 ts_tablet_manager.cc:1918] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:32:47.460435 16293 log.cc:1196] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/wal/wals/aa54d26157ab43468b2ae3ddcc3ddf66
I20260926 11:32:47.460767 16293 ts_tablet_manager.cc:1939] T aa54d26157ab43468b2ae3ddcc3ddf66 P 734e79c295e541f1911a62a82e8c42c8: Deleting consensus metadata
I20260926 11:32:47.461879 15780 catalog_manager.cc:4841] TS 734e79c295e541f1911a62a82e8c42c8 (127.0.98.3:42483): tablet aa54d26157ab43468b2ae3ddcc3ddf66 (table test-table [id=178835abd1464b4d8a4f6271b51f96a4]) successfully deleted
I20260926 11:32:47.463865 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 16124
I20260926 11:32:47.465121 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:32:47.464014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43335 (local address 127.0.0.1:47984)
0926 11:32:47.465000 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0926 11:32:47.465050 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":903}
I20260926 11:32:47.469132 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 15762
I20260926 11:33:07.476059 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44617
--webserver_interface=127.0.98.62
--webserver_port=40365
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:33:07.590322 16295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:07.590626 16295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:07.590668 16295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:07.594956 16295 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 11:33:07.595036 16295 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 11:33:07.595053 16295 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 11:33:07.595067 16295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:33:07.595088 16295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:07.595106 16295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:07.595124 16295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:33:07.595139 16295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:33:07.600817 16295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:07.601975 16295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:07.603046 16295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:33:07.617213 16302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:07.617336 16301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:07.617336 16304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:07.617393 16295 server_base.cc:1034] running on GCE node
I20260926 11:33:07.618229 16295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:07.619433 16295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:07.620625 16295 hybrid_clock.cc:648] HybridClock initialized: now 1790422387620578 us; error 67 us; skew 500 ppm
Sep 26 11:33:07 dist-test-slave-mrck krb5kdc[15749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:33:07.633389 16295 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:33:07.634589 16295 webserver.cc:460] Webserver started at http://127.0.98.62:40365/ using document root <none> and password file <none>
I20260926 11:33:07.635273 16295 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:07.635358 16295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:07.639635 16295 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:33:07.642104 16311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:07.643590 16295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 11:33:07.643812 16295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "4d9b63b3aa6645f49a00ad00664d0c09"
format_stamp: "Formatted at 2026-09-26 11:32:46 on dist-test-slave-mrck"
server_key: "dfe14d6a598c2a4601ddbf7f9c0ab01a"
server_key_iv: "617d94376fa144fc1b605ae895432b15"
server_key_version: "encryptionkey@0"
I20260926 11:33:07.644502 16295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:07.665797 16295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:07.669471 16295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:07.669690 16295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:07.677237 16295 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44617
I20260926 11:33:07.677517 16363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44617 every 8 connection(s)
I20260926 11:33:07.678467 16295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:33:07.682922 16364 sys_catalog.cc:263] Verifying existing consensus state
I20260926 11:33:07.683604 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 16295
I20260926 11:33:07.684134 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43321
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44617
--webserver_port=39597
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:33:07.685451 16364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09: Bootstrap starting.
I20260926 11:33:07.703212 16364 log.cc:824] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09: Log is configured to *not* fsync() on all Append() calls
I20260926 11:33:07.712337 16364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 11:33:07.712764 16364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09: Bootstrap complete.
I20260926 11:33:07.716976 16364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d9b63b3aa6645f49a00ad00664d0c09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44617 } }
I20260926 11:33:07.717386 16364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d9b63b3aa6645f49a00ad00664d0c09, State: Initialized, Role: FOLLOWER
I20260926 11:33:07.717953 16364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d9b63b3aa6645f49a00ad00664d0c09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44617 } }
I20260926 11:33:07.718080 16364 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:33:07.718180 16364 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:33:07.718286 16364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [term 1 FOLLOWER]: Advancing to term 2
I20260926 11:33:07.720383 16364 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d9b63b3aa6645f49a00ad00664d0c09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44617 } }
I20260926 11:33:07.720752 16364 leader_election.cc:302] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d9b63b3aa6645f49a00ad00664d0c09; no voters:
I20260926 11:33:07.721104 16364 leader_election.cc:288] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 11:33:07.721200 16368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 11:33:07.721408 16368 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [term 2 LEADER]: Becoming Leader. State: Replica: 4d9b63b3aa6645f49a00ad00664d0c09, State: Running, Role: LEADER
I20260926 11:33:07.721728 16368 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d9b63b3aa6645f49a00ad00664d0c09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44617 } }
I20260926 11:33:07.722405 16364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:33:07.723124 16369 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4d9b63b3aa6645f49a00ad00664d0c09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d9b63b3aa6645f49a00ad00664d0c09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44617 } } }
I20260926 11:33:07.723258 16369 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [sys.catalog]: This master's current role is: LEADER
I20260926 11:33:07.723930 16376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:33:07.723961 16370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d9b63b3aa6645f49a00ad00664d0c09. Latest consensus state: current_term: 2 leader_uuid: "4d9b63b3aa6645f49a00ad00664d0c09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d9b63b3aa6645f49a00ad00664d0c09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44617 } } }
I20260926 11:33:07.724087 16370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [sys.catalog]: This master's current role is: LEADER
I20260926 11:33:07.729110 16376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:33:07.729976 16376 catalog_manager.cc:1259] Loaded cluster ID: 065689bab73e4a5db2358bc164e61a5c
I20260926 11:33:07.730063 16376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:33:07.732734 16376 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:33:07.733446 16376 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09: Loaded TSK: 0
I20260926 11:33:07.748885 16376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09: Generated new TSK 1
I20260926 11:33:07.749215 16376 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 11:33:07.808233 16366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:07.808521 16366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:07.808566 16366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:07.812979 16366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:07.813048 16366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:07.813149 16366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:33:07.818578 16366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43321
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39597
--tserver_master_addrs=127.0.98.62:44617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:07.819665 16366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:07.820737 16366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:33:07.830860 16391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:07.830905 16392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:07.830905 16394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:07.831956 16366 server_base.cc:1034] running on GCE node
I20260926 11:33:07.832477 16366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:07.833351 16366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:07.834565 16366 hybrid_clock.cc:648] HybridClock initialized: now 1790422387834523 us; error 47 us; skew 500 ppm
Sep 26 11:33:07 dist-test-slave-mrck krb5kdc[15749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:33:07.838089 16366 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:33:07.839210 16366 webserver.cc:460] Webserver started at http://127.0.98.1:39597/ using document root <none> and password file <none>
I20260926 11:33:07.839902 16366 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:07.840073 16366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:07.844425 16366 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:33:07.846975 16401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:07.848577 16366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:33:07.848733 16366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "119a39a719be45f88987bfa5a856ace8"
format_stamp: "Formatted at 2026-09-26 11:32:46 on dist-test-slave-mrck"
server_key: "b0201dab8a58429f825ebe42dfa32cb1"
server_key_iv: "c091791fcf35001a00da506b2e31b5a3"
server_key_version: "encryptionkey@0"
I20260926 11:33:07.849453 16366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:07.868417 16366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:07.872082 16366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:07.872306 16366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:07.872993 16366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:33:07.874233 16408 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 11:33:07.878544 16366 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 11:33:07.878640 16366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260926 11:33:07.878728 16366 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 11:33:07.880493 16366 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 11:33:07.880569 16366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260926 11:33:07.880901 16408 tablet_bootstrap.cc:492] T aa54d26157ab43468b2ae3ddcc3ddf66 P 119a39a719be45f88987bfa5a856ace8: Bootstrap starting.
I20260926 11:33:07.891264 16366 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43321
I20260926 11:33:07.891345 16515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43321 every 8 connection(s)
I20260926 11:33:07.892412 16366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:33:07.901430 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 16366
I20260926 11:33:07.901922 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43335
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44617
--webserver_port=46795
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:33:07.901944 16408 log.cc:824] T aa54d26157ab43468b2ae3ddcc3ddf66 P 119a39a719be45f88987bfa5a856ace8: Log is configured to *not* fsync() on all Append() calls
Sep 26 11:33:07 dist-test-slave-mrck krb5kdc[15749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:33:07.911821 16408 tablet_bootstrap.cc:492] T aa54d26157ab43468b2ae3ddcc3ddf66 P 119a39a719be45f88987bfa5a856ace8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 11:33:07.912245 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:07.894840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45697 (local address 127.0.98.62:44617)
0926 11:33:07.895277 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:07.895297 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:07.895819 (+ 522us) server_negotiation.cc:407] Connection header received
0926 11:33:07.897437 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:07.897450 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:07.897803 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:07.898111 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:07.899902 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:07.900997 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:07.902631 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:07.903241 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:07.903415 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:07.906659 (+ 3244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:07.906665 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:07.906678 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:33:07.906710 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:07.908653 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:07.909338 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:07.909349 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:07.909355 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:07.909503 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:07.909875 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:07.909879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:07.909881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:07.910265 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:07.910420 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:07.911022 (+ 602us) server_negotiation.cc:299] Negotiation successful
0926 11:33:07.911313 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":147,"threads_started":1}
I20260926 11:33:07.912257 16519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:07.894848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.1:45697)
0926 11:33:07.895561 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:07.895632 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:07.896924 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:07.898274 (+ 1350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:07.898289 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:07.898856 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:33:07.899634 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:07.899679 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:07.901190 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:07.901194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:07.902491 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:07.902501 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:07.903643 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:07.903651 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:07.903922 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:07.904726 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:33:07.904756 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:33:07.906408 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:33:07.908806 (+ 2398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:33:07.908813 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:33:07.908827 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:33:07.909117 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:33:07.909623 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:33:07.909625 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:33:07.909627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:33:07.909758 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:33:07.910401 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:33:07.910408 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:33:07.910691 (+ 283us) client_negotiation.cc:769] Sending connection context
0926 11:33:07.910883 (+ 192us) client_negotiation.cc:240] Negotiation successful
0926 11:33:07.911246 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":139,"threads_started":1}
I20260926 11:33:07.912307 16408 tablet_bootstrap.cc:492] T aa54d26157ab43468b2ae3ddcc3ddf66 P 119a39a719be45f88987bfa5a856ace8: Bootstrap complete.
I20260926 11:33:07.913843 16408 ts_tablet_manager.cc:1397] T aa54d26157ab43468b2ae3ddcc3ddf66 P 119a39a719be45f88987bfa5a856ace8: Time spent bootstrapping tablet: real 0.033s user 0.026s sys 0.004s
I20260926 11:33:07.914965 16516 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44617
I20260926 11:33:07.915238 16516 heartbeater.cc:464] Registering TS with master...
I20260926 11:33:07.915831 16516 heartbeater.cc:511] Master 127.0.98.62:44617 requested a full tablet report, sending...
I20260926 11:33:07.917006 16408 raft_consensus.cc:348] T aa54d26157ab43468b2ae3ddcc3ddf66 P 119a39a719be45f88987bfa5a856ace8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a39a719be45f88987bfa5a856ace8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43321 } } peers { permanent_uuid: "734e79c295e541f1911a62a82e8c42c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42483 } } peers { permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 } }
I20260926 11:33:07.917521 16408 raft_consensus.cc:729] T aa54d26157ab43468b2ae3ddcc3ddf66 P 119a39a719be45f88987bfa5a856ace8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 119a39a719be45f88987bfa5a856ace8, State: Initialized, Role: FOLLOWER
I20260926 11:33:07.918018 16408 consensus_queue.cc:260] T aa54d26157ab43468b2ae3ddcc3ddf66 P 119a39a719be45f88987bfa5a856ace8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a39a719be45f88987bfa5a856ace8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43321 } } peers { permanent_uuid: "734e79c295e541f1911a62a82e8c42c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42483 } } peers { permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 } }
I20260926 11:33:07.918563 16328 ts_manager.cc:194] Registered new tserver with Master: 119a39a719be45f88987bfa5a856ace8 (127.0.98.1:43321)
I20260926 11:33:07.919051 16408 ts_tablet_manager.cc:1428] T aa54d26157ab43468b2ae3ddcc3ddf66 P 119a39a719be45f88987bfa5a856ace8: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260926 11:33:07.922237 16328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45697
I20260926 11:33:07.925269 16516 heartbeater.cc:503] Master 127.0.98.62:44617 was elected leader, sending a full tablet report...
Sep 26 11:33:07 dist-test-slave-mrck krb5kdc[15749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 11:33:07.933518 16524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:07.921939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38522 (local address 127.0.98.1:43321)
0926 11:33:07.923042 (+ 1103us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:07.923053 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:07.923076 (+ 23us) server_negotiation.cc:407] Connection header received
0926 11:33:07.923145 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:07.923150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:07.923357 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:07.923490 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:07.924398 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:07.925410 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:07.926539 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:07.927134 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:07.927709 (+ 575us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:07.929704 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:07.929714 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:07.929735 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:33:07.929760 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:07.931398 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:07.932008 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:07.932014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:07.932016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:07.932107 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:07.932407 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:07.932410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:07.932412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:07.932680 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:07.932829 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:07.933184 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 11:33:07.933356 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1005,"thread_start_us":103,"threads_started":1}
I20260926 11:33:07.933518 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:07.921578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43321 (local address 127.0.0.1:38522)
0926 11:33:07.922147 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:07.922202 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:07.922343 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:07.923530 (+ 1187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:07.923537 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:07.924057 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:33:07.924223 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:07.924234 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:07.925563 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:07.925566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:07.926394 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:07.926405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:07.927274 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:07.927293 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:07.927397 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:07.928090 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:33:07.928104 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:33:07.929527 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:33:07.931539 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:33:07.931550 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:33:07.931556 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:33:07.931881 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:33:07.932206 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:33:07.932210 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:33:07.932212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:33:07.932307 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:33:07.932808 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:33:07.932814 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:33:07.932948 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 11:33:07.933090 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 11:33:07.933322 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":133,"threads_started":1}
I20260926 11:33:07.934556 16450 tablet_service.cc:1514] Processing DeleteTablet for tablet aa54d26157ab43468b2ae3ddcc3ddf66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:32:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38522
I20260926 11:33:07.935441 16525 tablet_replica.cc:330] stopping tablet replica
I20260926 11:33:07.935796 16525 raft_consensus.cc:2229] T aa54d26157ab43468b2ae3ddcc3ddf66 P 119a39a719be45f88987bfa5a856ace8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:33:07.936067 16525 raft_consensus.cc:2258] T aa54d26157ab43468b2ae3ddcc3ddf66 P 119a39a719be45f88987bfa5a856ace8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:33:07.937040 16525 ts_tablet_manager.cc:1905] T aa54d26157ab43468b2ae3ddcc3ddf66 P 119a39a719be45f88987bfa5a856ace8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:33:07.940712 16525 ts_tablet_manager.cc:1918] T aa54d26157ab43468b2ae3ddcc3ddf66 P 119a39a719be45f88987bfa5a856ace8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:33:07.940810 16525 log.cc:1196] T aa54d26157ab43468b2ae3ddcc3ddf66 P 119a39a719be45f88987bfa5a856ace8: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/wal/wals/aa54d26157ab43468b2ae3ddcc3ddf66
I20260926 11:33:07.941107 16525 ts_tablet_manager.cc:1939] T aa54d26157ab43468b2ae3ddcc3ddf66 P 119a39a719be45f88987bfa5a856ace8: Deleting consensus metadata
I20260926 11:33:07.942919 16314 catalog_manager.cc:4841] TS 119a39a719be45f88987bfa5a856ace8 (127.0.98.1:43321): tablet aa54d26157ab43468b2ae3ddcc3ddf66 (table test-table [id=178835abd1464b4d8a4f6271b51f96a4]) successfully deleted
W20260926 11:33:08.024108 16521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:08.024511 16521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:08.024596 16521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:08.028806 16521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:08.028903 16521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:08.028998 16521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:33:08.034493 16521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43335
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46795
--tserver_master_addrs=127.0.98.62:44617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:08.035650 16521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:08.036739 16521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:33:08.046907 16530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:08.046943 16533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:08.047027 16521 server_base.cc:1034] running on GCE node
W20260926 11:33:08.046903 16531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:08.047781 16521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:08.048543 16521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:08.049791 16521 hybrid_clock.cc:648] HybridClock initialized: now 1790422388049773 us; error 33 us; skew 500 ppm
Sep 26 11:33:08 dist-test-slave-mrck krb5kdc[15749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:33:08.052912 16521 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:33:08.054030 16521 webserver.cc:460] Webserver started at http://127.0.98.2:46795/ using document root <none> and password file <none>
I20260926 11:33:08.054719 16521 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:08.054848 16521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:08.059230 16521 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 11:33:08.061676 16540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:08.063031 16521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 11:33:08.063186 16521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "9efbdbfb4d664f32a69fe60963885f21"
format_stamp: "Formatted at 2026-09-26 11:32:46 on dist-test-slave-mrck"
server_key: "a7085119b68a3212b78d9c5abdda1472"
server_key_iv: "dd5f76d2b089eb8e2225aaf35cda8115"
server_key_version: "encryptionkey@0"
I20260926 11:33:08.063793 16521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:08.083863 16521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:08.087577 16521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:08.087877 16521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:08.088573 16521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:33:08.089797 16547 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 11:33:08.094197 16521 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 11:33:08.094321 16521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260926 11:33:08.094421 16521 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 11:33:08.096328 16521 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 11:33:08.096407 16521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260926 11:33:08.096727 16547 tablet_bootstrap.cc:492] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21: Bootstrap starting.
I20260926 11:33:08.103782 16547 log.cc:824] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21: Log is configured to *not* fsync() on all Append() calls
I20260926 11:33:08.106936 16521 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43335
I20260926 11:33:08.106964 16655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43335 every 8 connection(s)
I20260926 11:33:08.107950 16521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:33:08.110167 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 16521
I20260926 11:33:08.110653 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42483
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44617
--webserver_port=36973
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:33:08.113786 16547 tablet_bootstrap.cc:492] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 11:33:08.114275 16547 tablet_bootstrap.cc:492] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21: Bootstrap complete.
I20260926 11:33:08.114785 16547 ts_tablet_manager.cc:1397] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.004s
I20260926 11:33:08.117471 16547 raft_consensus.cc:348] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a39a719be45f88987bfa5a856ace8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43321 } } peers { permanent_uuid: "734e79c295e541f1911a62a82e8c42c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42483 } } peers { permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 } }
I20260926 11:33:08.118064 16547 raft_consensus.cc:729] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9efbdbfb4d664f32a69fe60963885f21, State: Initialized, Role: FOLLOWER
I20260926 11:33:08.118610 16547 consensus_queue.cc:260] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "119a39a719be45f88987bfa5a856ace8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43321 } } peers { permanent_uuid: "734e79c295e541f1911a62a82e8c42c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42483 } } peers { permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 } }
I20260926 11:33:08.119611 16547 ts_tablet_manager.cc:1428] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
Sep 26 11:33:08 dist-test-slave-mrck krb5kdc[15749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:33:08.125541 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:08.109936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46901 (local address 127.0.98.62:44617)
0926 11:33:08.110100 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:08.110105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:08.110878 (+ 773us) server_negotiation.cc:407] Connection header received
0926 11:33:08.112186 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:08.112191 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:08.112260 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:08.112320 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:08.114224 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:08.115151 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:08.117248 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:08.117607 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:08.117734 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:08.121286 (+ 3552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:08.121295 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:08.121297 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:33:08.121326 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:08.122964 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:08.123721 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:08.123765 (+ 44us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:08.123768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:08.123871 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:08.124263 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:08.124265 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:08.124266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:08.124497 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:08.124604 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:08.125289 (+ 685us) server_negotiation.cc:299] Negotiation successful
0926 11:33:08.125413 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 11:33:08.126627 16658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:08.110089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.2:46901)
0926 11:33:08.110599 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:08.110653 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:08.111907 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:08.112753 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:08.112765 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:08.113301 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:33:08.114030 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:08.114054 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:08.115346 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:08.115352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:08.117066 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:08.117085 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:08.117771 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:08.117778 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:08.118035 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:08.119264 (+ 1229us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:33:08.119291 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:33:08.121077 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:33:08.123115 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:33:08.123122 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:33:08.123141 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:33:08.123559 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:33:08.123966 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:33:08.123969 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:33:08.123971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:33:08.124160 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:33:08.124608 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:33:08.124615 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:33:08.124954 (+ 339us) client_negotiation.cc:769] Sending connection context
0926 11:33:08.125175 (+ 221us) client_negotiation.cc:240] Negotiation successful
0926 11:33:08.125534 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":155,"threads_started":1}
I20260926 11:33:08.128052 16656 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44617
I20260926 11:33:08.128356 16656 heartbeater.cc:464] Registering TS with master...
I20260926 11:33:08.128929 16656 heartbeater.cc:511] Master 127.0.98.62:44617 requested a full tablet report, sending...
I20260926 11:33:08.130690 16328 ts_manager.cc:194] Registered new tserver with Master: 9efbdbfb4d664f32a69fe60963885f21 (127.0.98.2:43335)
I20260926 11:33:08.131922 16328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46901
I20260926 11:33:08.134128 16656 heartbeater.cc:503] Master 127.0.98.62:44617 was elected leader, sending a full tablet report...
Sep 26 11:33:08 dist-test-slave-mrck krb5kdc[15749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260926 11:33:08.141388 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:08.131596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43335 (local address 127.0.0.1:58290)
0926 11:33:08.131802 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:08.131817 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:08.131896 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:08.132636 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:08.132640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:08.132840 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:33:08.132942 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:08.132949 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:08.134420 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:08.134423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:08.135091 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:08.135100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:08.135904 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:08.135920 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:08.136016 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:08.136527 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:33:08.136539 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:33:08.137893 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:33:08.139773 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:33:08.139776 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:33:08.139778 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:33:08.140027 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:33:08.140319 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:33:08.140322 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:33:08.140324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:33:08.140418 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:33:08.140845 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:33:08.140848 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:33:08.140955 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 11:33:08.141063 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 11:33:08.141216 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260926 11:33:08.141409 16661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:08.131717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58290 (local address 127.0.98.2:43335)
0926 11:33:08.132134 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:08.132145 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:08.132166 (+ 21us) server_negotiation.cc:407] Connection header received
0926 11:33:08.132234 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:08.132240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:08.132454 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:08.132633 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:08.133069 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:08.134287 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:08.135215 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:08.135809 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:08.135916 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:08.138045 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:08.138053 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:08.138064 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:33:08.138088 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:08.139632 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:08.140137 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:08.140142 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:08.140143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:08.140225 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:08.140505 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:08.140508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:08.140509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:08.140741 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:08.140857 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:08.141135 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 11:33:08.141273 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":140,"threads_started":1}
I20260926 11:33:08.142629 16589 tablet_service.cc:1514] Processing DeleteTablet for tablet aa54d26157ab43468b2ae3ddcc3ddf66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:32:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58290
I20260926 11:33:08.143721 16662 tablet_replica.cc:330] stopping tablet replica
I20260926 11:33:08.144106 16662 raft_consensus.cc:2229] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:33:08.144366 16662 raft_consensus.cc:2258] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:33:08.145354 16662 ts_tablet_manager.cc:1905] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:33:08.149497 16662 ts_tablet_manager.cc:1918] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:33:08.149631 16662 log.cc:1196] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/wal/wals/aa54d26157ab43468b2ae3ddcc3ddf66
I20260926 11:33:08.149916 16662 ts_tablet_manager.cc:1939] T aa54d26157ab43468b2ae3ddcc3ddf66 P 9efbdbfb4d664f32a69fe60963885f21: Deleting consensus metadata
I20260926 11:33:08.152032 16313 catalog_manager.cc:4841] TS 9efbdbfb4d664f32a69fe60963885f21 (127.0.98.2:43335): tablet aa54d26157ab43468b2ae3ddcc3ddf66 (table test-table [id=178835abd1464b4d8a4f6271b51f96a4]) successfully deleted
W20260926 11:33:08.234548 16659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:08.234830 16659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:08.234906 16659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:08.239112 16659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:08.239185 16659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:08.239290 16659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:33:08.244936 16659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36973
--tserver_master_addrs=127.0.98.62:44617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:08.246127 16659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:08.247280 16659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:33:08.257688 16670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:08.257733 16668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:08.257726 16667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:08.257978 16659 server_base.cc:1034] running on GCE node
I20260926 11:33:08.258476 16659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:08.259162 16659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:08.260346 16659 hybrid_clock.cc:648] HybridClock initialized: now 1790422388260305 us; error 59 us; skew 500 ppm
Sep 26 11:33:08 dist-test-slave-mrck krb5kdc[15749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:33:08.263160 16659 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:33:08.264139 16659 webserver.cc:460] Webserver started at http://127.0.98.3:36973/ using document root <none> and password file <none>
I20260926 11:33:08.264737 16659 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:08.264843 16659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:08.268983 16659 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:33:08.271090 16677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:08.272521 16659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 11:33:08.272662 16659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "734e79c295e541f1911a62a82e8c42c8"
format_stamp: "Formatted at 2026-09-26 11:32:47 on dist-test-slave-mrck"
server_key: "c1e319c1370928ee0f125e9a8317fef0"
server_key_iv: "c0f9b0b6715c8979445be1ae5b83570e"
server_key_version: "encryptionkey@0"
I20260926 11:33:08.273257 16659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:08.292954 16659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:08.296628 16659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:08.296855 16659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:08.297520 16659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:33:08.298460 16659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:33:08.298530 16659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:08.298605 16659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:33:08.298656 16659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:08.308064 16659 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42483
I20260926 11:33:08.308104 16790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42483 every 8 connection(s)
I20260926 11:33:08.309203 16659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:33:08.318753 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 16659
Sep 26 11:33:08 dist-test-slave-mrck krb5kdc[15749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422388, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:33:08.325197 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:08.311293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57031 (local address 127.0.98.62:44617)
0926 11:33:08.311455 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:08.311459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:08.312396 (+ 937us) server_negotiation.cc:407] Connection header received
0926 11:33:08.313779 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:08.313783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:08.313852 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:08.313975 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:08.315301 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:08.315920 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:08.317210 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:08.317453 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:08.317578 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:08.320589 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:08.320592 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:08.320595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:33:08.320623 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:08.322355 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:08.323261 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:08.323265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:08.323267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:08.323378 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:08.323820 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:08.323824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:08.323825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:08.324074 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:08.324206 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:08.324956 (+ 750us) server_negotiation.cc:299] Negotiation successful
0926 11:33:08.325069 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 11:33:08.326490 16793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:08.311616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.3:57031)
0926 11:33:08.312215 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:08.312262 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:08.313517 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:08.314185 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:08.314193 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:08.314604 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:33:08.315119 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:08.315134 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:08.316084 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:08.316088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:08.317057 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:08.317068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:08.317647 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:08.317652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:08.317838 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:08.318522 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:33:08.318540 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:33:08.320404 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:33:08.322553 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:33:08.322559 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:33:08.322574 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:33:08.322993 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:33:08.323475 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:33:08.323479 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:33:08.323481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:33:08.323689 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:33:08.324242 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:33:08.324250 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:33:08.324634 (+ 384us) client_negotiation.cc:769] Sending connection context
0926 11:33:08.324890 (+ 256us) client_negotiation.cc:240] Negotiation successful
0926 11:33:08.325305 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":139,"threads_started":1}
I20260926 11:33:08.328001 16791 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44617
I20260926 11:33:08.328286 16791 heartbeater.cc:464] Registering TS with master...
I20260926 11:33:08.328913 16791 heartbeater.cc:511] Master 127.0.98.62:44617 requested a full tablet report, sending...
I20260926 11:33:08.329686 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:08.319548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56650 (local address 127.0.98.62:44617)
0926 11:33:08.319892 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:08.319895 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:08.320571 (+ 676us) server_negotiation.cc:407] Connection header received
0926 11:33:08.320603 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:08.320607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:08.320673 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:08.320766 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:08.321679 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:08.322568 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:08.323795 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:08.324133 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:08.324250 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:08.325799 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:08.325802 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:08.325804 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:33:08.325825 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:08.327873 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:08.328414 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:08.328418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:08.328420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:08.328508 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:08.328792 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:08.328794 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:08.328795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:08.328988 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:08.329174 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:08.329391 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 11:33:08.329514 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":94,"threads_started":1}
I20260926 11:33:08.330116 16328 ts_manager.cc:194] Registered new tserver with Master: 734e79c295e541f1911a62a82e8c42c8 (127.0.98.3:42483)
I20260926 11:33:08.330806 16328 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57031
I20260926 11:33:08.331406 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:33:08.335526 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:08.332033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56654 (local address 127.0.98.62:44617)
0926 11:33:08.332154 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:08.332158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:08.332360 (+ 202us) server_negotiation.cc:407] Connection header received
0926 11:33:08.332534 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:08.332537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:08.332634 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:08.332737 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:33:08.332973 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:08.333480 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:08.334338 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:08.334584 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:08.334695 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:08.335001 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:33:08.335134 (+ 133us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 11:33:08.335398 (+ 264us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56654: token expired
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 11:33:08.335623 16795 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56654: token expired
W20260926 11:33:08.335970 16259 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:44617)
I20260926 11:33:08.343852 16795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:08.336419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56664 (local address 127.0.98.62:44617)
0926 11:33:08.336542 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:08.336546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:08.336561 (+ 15us) server_negotiation.cc:407] Connection header received
0926 11:33:08.336661 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:08.336664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:08.336712 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:08.336794 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:08.337434 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:08.337948 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:08.338766 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:08.339000 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:08.339109 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:08.340637 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:08.340641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:08.340644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:33:08.340680 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:08.341972 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:08.342454 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:08.342459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:08.342461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:08.342562 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:08.342874 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:08.342880 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:08.342881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:08.343090 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:08.343239 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:08.343503 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 11:33:08.343642 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 11:33:08.346547 16326 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:33:08.350121 16326 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:33:08.357795 16589 tablet_service.cc:1467] Processing CreateTablet for tablet 4378340ddf0942f6a28ee77b11b798c6 (DEFAULT_TABLE table=test-table [id=bc3ed7211ab249b986fc66ada717b1ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:33:08.358011 16450 tablet_service.cc:1467] Processing CreateTablet for tablet 4378340ddf0942f6a28ee77b11b798c6 (DEFAULT_TABLE table=test-table [id=bc3ed7211ab249b986fc66ada717b1ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:33:08.358465 16589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4378340ddf0942f6a28ee77b11b798c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:33:08.358598 16450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4378340ddf0942f6a28ee77b11b798c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:33:08.361238 16408 tablet_bootstrap.cc:492] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8: Bootstrap starting.
I20260926 11:33:08.361562 16547 tablet_bootstrap.cc:492] T 4378340ddf0942f6a28ee77b11b798c6 P 9efbdbfb4d664f32a69fe60963885f21: Bootstrap starting.
I20260926 11:33:08.362082 16408 tablet_bootstrap.cc:654] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8: Neither blocks nor log segments found. Creating new log.
I20260926 11:33:08.362505 16547 tablet_bootstrap.cc:654] T 4378340ddf0942f6a28ee77b11b798c6 P 9efbdbfb4d664f32a69fe60963885f21: Neither blocks nor log segments found. Creating new log.
I20260926 11:33:08.363196 16523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:08.357013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42483 (local address 127.0.0.1:52374)
0926 11:33:08.357256 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:08.357272 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:08.357350 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:08.358286 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:08.358289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:08.358307 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:08.358423 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:08.358429 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:08.360096 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:08.360100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:08.361340 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:08.361352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:08.362770 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:08.362774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:08.362891 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:08.362946 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 11:33:08.363023 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 11:33:08.363089 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260926 11:33:08.363375 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:08.357268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52374 (local address 127.0.98.3:42483)
0926 11:33:08.357699 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:08.357716 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:08.357741 (+ 25us) server_negotiation.cc:407] Connection header received
0926 11:33:08.357818 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:08.357824 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:08.358089 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:08.358587 (+ 498us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:08.358788 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:08.359952 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:08.361698 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:08.362610 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:08.362734 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:08.362948 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:08.363097 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 11:33:08.363221 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":215,"threads_started":1}
I20260926 11:33:08.364516 16547 tablet_bootstrap.cc:492] T 4378340ddf0942f6a28ee77b11b798c6 P 9efbdbfb4d664f32a69fe60963885f21: No bootstrap required, opened a new log
I20260926 11:33:08.364626 16547 ts_tablet_manager.cc:1397] T 4378340ddf0942f6a28ee77b11b798c6 P 9efbdbfb4d664f32a69fe60963885f21: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 11:33:08.364794 16408 tablet_bootstrap.cc:492] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8: No bootstrap required, opened a new log
I20260926 11:33:08.364902 16408 ts_tablet_manager.cc:1397] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 11:33:08.365120 16547 raft_consensus.cc:348] T 4378340ddf0942f6a28ee77b11b798c6 P 9efbdbfb4d664f32a69fe60963885f21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734e79c295e541f1911a62a82e8c42c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42483 } } peers { permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 } } peers { permanent_uuid: "119a39a719be45f88987bfa5a856ace8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43321 } }
I20260926 11:33:08.365250 16547 raft_consensus.cc:374] T 4378340ddf0942f6a28ee77b11b798c6 P 9efbdbfb4d664f32a69fe60963885f21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:33:08.365296 16547 raft_consensus.cc:729] T 4378340ddf0942f6a28ee77b11b798c6 P 9efbdbfb4d664f32a69fe60963885f21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9efbdbfb4d664f32a69fe60963885f21, State: Initialized, Role: FOLLOWER
I20260926 11:33:08.365321 16408 raft_consensus.cc:348] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734e79c295e541f1911a62a82e8c42c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42483 } } peers { permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 } } peers { permanent_uuid: "119a39a719be45f88987bfa5a856ace8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43321 } }
I20260926 11:33:08.365442 16408 raft_consensus.cc:374] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:33:08.365484 16408 raft_consensus.cc:729] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 119a39a719be45f88987bfa5a856ace8, State: Initialized, Role: FOLLOWER
I20260926 11:33:08.365429 16547 consensus_queue.cc:260] T 4378340ddf0942f6a28ee77b11b798c6 P 9efbdbfb4d664f32a69fe60963885f21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734e79c295e541f1911a62a82e8c42c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42483 } } peers { permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 } } peers { permanent_uuid: "119a39a719be45f88987bfa5a856ace8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43321 } }
I20260926 11:33:08.365608 16408 consensus_queue.cc:260] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734e79c295e541f1911a62a82e8c42c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42483 } } peers { permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 } } peers { permanent_uuid: "119a39a719be45f88987bfa5a856ace8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43321 } }
I20260926 11:33:08.365736 16547 ts_tablet_manager.cc:1428] T 4378340ddf0942f6a28ee77b11b798c6 P 9efbdbfb4d664f32a69fe60963885f21: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 11:33:08.365867 16408 ts_tablet_manager.cc:1428] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 11:33:08.366842 16725 tablet_service.cc:1467] Processing CreateTablet for tablet 4378340ddf0942f6a28ee77b11b798c6 (DEFAULT_TABLE table=test-table [id=bc3ed7211ab249b986fc66ada717b1ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:33:08.367981 16725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4378340ddf0942f6a28ee77b11b798c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:33:08.374774 16803 tablet_bootstrap.cc:492] T 4378340ddf0942f6a28ee77b11b798c6 P 734e79c295e541f1911a62a82e8c42c8: Bootstrap starting.
I20260926 11:33:08.377480 16803 tablet_bootstrap.cc:654] T 4378340ddf0942f6a28ee77b11b798c6 P 734e79c295e541f1911a62a82e8c42c8: Neither blocks nor log segments found. Creating new log.
I20260926 11:33:08.378253 16803 log.cc:824] T 4378340ddf0942f6a28ee77b11b798c6 P 734e79c295e541f1911a62a82e8c42c8: Log is configured to *not* fsync() on all Append() calls
I20260926 11:33:08.380457 16803 tablet_bootstrap.cc:492] T 4378340ddf0942f6a28ee77b11b798c6 P 734e79c295e541f1911a62a82e8c42c8: No bootstrap required, opened a new log
I20260926 11:33:08.380676 16803 ts_tablet_manager.cc:1397] T 4378340ddf0942f6a28ee77b11b798c6 P 734e79c295e541f1911a62a82e8c42c8: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260926 11:33:08.383998 16803 raft_consensus.cc:348] T 4378340ddf0942f6a28ee77b11b798c6 P 734e79c295e541f1911a62a82e8c42c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734e79c295e541f1911a62a82e8c42c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42483 } } peers { permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 } } peers { permanent_uuid: "119a39a719be45f88987bfa5a856ace8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43321 } }
I20260926 11:33:08.384284 16803 raft_consensus.cc:374] T 4378340ddf0942f6a28ee77b11b798c6 P 734e79c295e541f1911a62a82e8c42c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:33:08.384361 16803 raft_consensus.cc:729] T 4378340ddf0942f6a28ee77b11b798c6 P 734e79c295e541f1911a62a82e8c42c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 734e79c295e541f1911a62a82e8c42c8, State: Initialized, Role: FOLLOWER
I20260926 11:33:08.384805 16803 consensus_queue.cc:260] T 4378340ddf0942f6a28ee77b11b798c6 P 734e79c295e541f1911a62a82e8c42c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734e79c295e541f1911a62a82e8c42c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42483 } } peers { permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 } } peers { permanent_uuid: "119a39a719be45f88987bfa5a856ace8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43321 } }
I20260926 11:33:08.385519 16791 heartbeater.cc:503] Master 127.0.98.62:44617 was elected leader, sending a full tablet report...
I20260926 11:33:08.385748 16803 ts_tablet_manager.cc:1428] T 4378340ddf0942f6a28ee77b11b798c6 P 734e79c295e541f1911a62a82e8c42c8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260926 11:33:08.393659 16517 tablet.cc:2369] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 11:33:08.560262 16792 tablet.cc:2369] T 4378340ddf0942f6a28ee77b11b798c6 P 734e79c295e541f1911a62a82e8c42c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:33:08.579566 16522 raft_consensus.cc:482] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:33:08.579819 16522 raft_consensus.cc:504] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734e79c295e541f1911a62a82e8c42c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42483 } } peers { permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 } } peers { permanent_uuid: "119a39a719be45f88987bfa5a856ace8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43321 } }
I20260926 11:33:08.581002 16522 leader_election.cc:288] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 734e79c295e541f1911a62a82e8c42c8 (127.0.98.3:42483), 9efbdbfb4d664f32a69fe60963885f21 (127.0.98.2:43335)
I20260926 11:33:08.586972 16800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:08.581436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54941 (local address 127.0.98.3:42483)
0926 11:33:08.581599 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:08.581603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:08.581869 (+ 266us) server_negotiation.cc:407] Connection header received
0926 11:33:08.582067 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:08.582070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:08.582125 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:08.582249 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:08.582642 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:08.583452 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:08.585411 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:08.586041 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:08.586174 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:08.586275 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:08.586746 (+ 471us) server_negotiation.cc:299] Negotiation successful
0926 11:33:08.586839 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 11:33:08.586972 16661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:08.581534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57471 (local address 127.0.98.2:43335)
0926 11:33:08.581665 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:08.581668 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:08.581936 (+ 268us) server_negotiation.cc:407] Connection header received
0926 11:33:08.582067 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:08.582070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:08.582139 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:08.582237 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:08.582642 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:08.583447 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:08.585411 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:08.586106 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:08.586208 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:08.586370 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:08.586751 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 11:33:08.586835 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 11:33:08.587044 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:08.581315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42483 (local address 127.0.98.1:54941)
0926 11:33:08.581762 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:08.581779 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:08.581921 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:08.582278 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:08.582281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:08.582326 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:08.582521 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:08.582527 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:08.583587 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:08.583591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:08.585255 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:08.585266 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:08.586243 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:08.586247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:08.586503 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:08.586624 (+ 121us) client_negotiation.cc:769] Sending connection context
0926 11:33:08.586706 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 11:33:08.586836 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":356,"spinlock_wait_cycles":67328,"thread_start_us":106,"threads_started":1}
I20260926 11:33:08.587056 16807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:08.581315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43335 (local address 127.0.98.1:57471)
0926 11:33:08.581849 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:08.581862 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:08.581934 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:08.582278 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:08.582281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:08.582327 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:08.582508 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:08.582518 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:08.583587 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:08.583591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:08.585255 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:08.585266 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:08.586243 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:08.586247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:08.586525 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:08.586617 (+ 92us) client_negotiation.cc:769] Sending connection context
0926 11:33:08.586706 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 11:33:08.586817 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":478,"spinlock_wait_cycles":9984,"thread_start_us":55,"threads_started":1}
I20260926 11:33:08.587774 16609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4378340ddf0942f6a28ee77b11b798c6" candidate_uuid: "119a39a719be45f88987bfa5a856ace8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efbdbfb4d664f32a69fe60963885f21" is_pre_election: true
I20260926 11:33:08.587857 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4378340ddf0942f6a28ee77b11b798c6" candidate_uuid: "119a39a719be45f88987bfa5a856ace8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734e79c295e541f1911a62a82e8c42c8" is_pre_election: true
I20260926 11:33:08.588102 16609 raft_consensus.cc:2454] T 4378340ddf0942f6a28ee77b11b798c6 P 9efbdbfb4d664f32a69fe60963885f21 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 119a39a719be45f88987bfa5a856ace8 in term 0.
I20260926 11:33:08.588147 16745 raft_consensus.cc:2454] T 4378340ddf0942f6a28ee77b11b798c6 P 734e79c295e541f1911a62a82e8c42c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 119a39a719be45f88987bfa5a856ace8 in term 0.
I20260926 11:33:08.588647 16403 leader_election.cc:302] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 119a39a719be45f88987bfa5a856ace8, 734e79c295e541f1911a62a82e8c42c8, 9efbdbfb4d664f32a69fe60963885f21; no voters:
I20260926 11:33:08.589020 16522 raft_consensus.cc:2781] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:33:08.589157 16522 raft_consensus.cc:482] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:33:08.589229 16522 raft_consensus.cc:3037] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:33:08.590305 16522 raft_consensus.cc:504] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734e79c295e541f1911a62a82e8c42c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42483 } } peers { permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 } } peers { permanent_uuid: "119a39a719be45f88987bfa5a856ace8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43321 } }
I20260926 11:33:08.590699 16522 leader_election.cc:288] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [CANDIDATE]: Term 1 election: Requested vote from peers 734e79c295e541f1911a62a82e8c42c8 (127.0.98.3:42483), 9efbdbfb4d664f32a69fe60963885f21 (127.0.98.2:43335)
I20260926 11:33:08.591118 16745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4378340ddf0942f6a28ee77b11b798c6" candidate_uuid: "119a39a719be45f88987bfa5a856ace8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "734e79c295e541f1911a62a82e8c42c8"
I20260926 11:33:08.591209 16609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4378340ddf0942f6a28ee77b11b798c6" candidate_uuid: "119a39a719be45f88987bfa5a856ace8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efbdbfb4d664f32a69fe60963885f21"
I20260926 11:33:08.591281 16745 raft_consensus.cc:3037] T 4378340ddf0942f6a28ee77b11b798c6 P 734e79c295e541f1911a62a82e8c42c8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:33:08.591339 16609 raft_consensus.cc:3037] T 4378340ddf0942f6a28ee77b11b798c6 P 9efbdbfb4d664f32a69fe60963885f21 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:33:08.592643 16609 raft_consensus.cc:2454] T 4378340ddf0942f6a28ee77b11b798c6 P 9efbdbfb4d664f32a69fe60963885f21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 119a39a719be45f88987bfa5a856ace8 in term 1.
I20260926 11:33:08.592643 16745 raft_consensus.cc:2454] T 4378340ddf0942f6a28ee77b11b798c6 P 734e79c295e541f1911a62a82e8c42c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 119a39a719be45f88987bfa5a856ace8 in term 1.
I20260926 11:33:08.593006 16404 leader_election.cc:302] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 119a39a719be45f88987bfa5a856ace8, 734e79c295e541f1911a62a82e8c42c8, 9efbdbfb4d664f32a69fe60963885f21; no voters:
I20260926 11:33:08.593204 16522 raft_consensus.cc:2781] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:33:08.593416 16522 raft_consensus.cc:686] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [term 1 LEADER]: Becoming Leader. State: Replica: 119a39a719be45f88987bfa5a856ace8, State: Running, Role: LEADER
I20260926 11:33:08.593698 16522 consensus_queue.cc:237] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734e79c295e541f1911a62a82e8c42c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42483 } } peers { permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 } } peers { permanent_uuid: "119a39a719be45f88987bfa5a856ace8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43321 } }
I20260926 11:33:08.596591 16328 catalog_manager.cc:5495] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 119a39a719be45f88987bfa5a856ace8 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "119a39a719be45f88987bfa5a856ace8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734e79c295e541f1911a62a82e8c42c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "119a39a719be45f88987bfa5a856ace8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43321 } health_report { overall_health: HEALTHY } } }
W20260926 11:33:08.609206 16657 tablet.cc:2369] T 4378340ddf0942f6a28ee77b11b798c6 P 9efbdbfb4d664f32a69fe60963885f21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:33:08.614506 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:08.611024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38534 (local address 127.0.98.1:43321)
0926 11:33:08.611293 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:08.611300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:08.611313 (+ 13us) server_negotiation.cc:407] Connection header received
0926 11:33:08.611387 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:08.611389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:08.611465 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:08.611600 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:33:08.611874 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:08.612378 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:08.613189 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:08.613402 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:08.613480 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:08.613733 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:33:08.614170 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:33:08.614258 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:08.614329 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 11:33:08.614376 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":117,"threads_started":1}
I20260926 11:33:08.618629 16745 raft_consensus.cc:1261] T 4378340ddf0942f6a28ee77b11b798c6 P 734e79c295e541f1911a62a82e8c42c8 [term 1 FOLLOWER]: Refusing update from remote peer 119a39a719be45f88987bfa5a856ace8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:33:08.618626 16609 raft_consensus.cc:1261] T 4378340ddf0942f6a28ee77b11b798c6 P 9efbdbfb4d664f32a69fe60963885f21 [term 1 FOLLOWER]: Refusing update from remote peer 119a39a719be45f88987bfa5a856ace8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:33:08.619416 16808 consensus_queue.cc:1035] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:33:08.619647 16522 consensus_queue.cc:1035] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "734e79c295e541f1911a62a82e8c42c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:33:08.627151 16813 mvcc.cc:205] Tried to move back new op lower bound from 7333570103774490624 to 7333570103682252800. Current Snapshot: MvccSnapshot[applied={T|T < 7333570103774490624}]
I20260926 11:33:08.634519 16328 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:33:08.634677 16328 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:33:08.637092 16328 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09: Sending DeleteTablet for 3 replicas of tablet 4378340ddf0942f6a28ee77b11b798c6
I20260926 11:33:08.637880 16589 tablet_service.cc:1514] Processing DeleteTablet for tablet 4378340ddf0942f6a28ee77b11b798c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:33:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58290
I20260926 11:33:08.637894 16450 tablet_service.cc:1514] Processing DeleteTablet for tablet 4378340ddf0942f6a28ee77b11b798c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:33:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38522
I20260926 11:33:08.638074 16662 tablet_replica.cc:330] T 4378340ddf0942f6a28ee77b11b798c6 P 9efbdbfb4d664f32a69fe60963885f21: stopping tablet replica
I20260926 11:33:08.638187 16662 raft_consensus.cc:2229] T 4378340ddf0942f6a28ee77b11b798c6 P 9efbdbfb4d664f32a69fe60963885f21 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:33:08.638279 16822 tablet_replica.cc:330] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8: stopping tablet replica
I20260926 11:33:08.638309 16662 raft_consensus.cc:2258] T 4378340ddf0942f6a28ee77b11b798c6 P 9efbdbfb4d664f32a69fe60963885f21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:33:08.638264 16725 tablet_service.cc:1514] Processing DeleteTablet for tablet 4378340ddf0942f6a28ee77b11b798c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:33:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52374
I20260926 11:33:08.638389 16822 raft_consensus.cc:2229] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:33:08.638698 16822 raft_consensus.cc:2258] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:33:08.638828 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 16366
I20260926 11:33:08.639827 16662 ts_tablet_manager.cc:1905] T 4378340ddf0942f6a28ee77b11b798c6 P 9efbdbfb4d664f32a69fe60963885f21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:33:08.640761 16823 tablet_replica.cc:330] T 4378340ddf0942f6a28ee77b11b798c6 P 734e79c295e541f1911a62a82e8c42c8: stopping tablet replica
I20260926 11:33:08.641144 16823 raft_consensus.cc:2229] T 4378340ddf0942f6a28ee77b11b798c6 P 734e79c295e541f1911a62a82e8c42c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:33:08.641503 16823 raft_consensus.cc:2258] T 4378340ddf0942f6a28ee77b11b798c6 P 734e79c295e541f1911a62a82e8c42c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:33:08.642372 16662 ts_tablet_manager.cc:1918] T 4378340ddf0942f6a28ee77b11b798c6 P 9efbdbfb4d664f32a69fe60963885f21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:33:08.642470 16662 log.cc:1196] T 4378340ddf0942f6a28ee77b11b798c6 P 9efbdbfb4d664f32a69fe60963885f21: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/wal/wals/4378340ddf0942f6a28ee77b11b798c6
I20260926 11:33:08.642795 16662 ts_tablet_manager.cc:1939] T 4378340ddf0942f6a28ee77b11b798c6 P 9efbdbfb4d664f32a69fe60963885f21: Deleting consensus metadata
I20260926 11:33:08.643239 16823 ts_tablet_manager.cc:1905] T 4378340ddf0942f6a28ee77b11b798c6 P 734e79c295e541f1911a62a82e8c42c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:33:08.643718 16313 catalog_manager.cc:4841] TS 9efbdbfb4d664f32a69fe60963885f21 (127.0.98.2:43335): tablet 4378340ddf0942f6a28ee77b11b798c6 (table test-table [id=bc3ed7211ab249b986fc66ada717b1ab]) successfully deleted
W20260926 11:33:08.646431 16314 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43321 (error 108)
I20260926 11:33:08.646809 16823 ts_tablet_manager.cc:1918] T 4378340ddf0942f6a28ee77b11b798c6 P 734e79c295e541f1911a62a82e8c42c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:33:08.646976 16823 log.cc:1196] T 4378340ddf0942f6a28ee77b11b798c6 P 734e79c295e541f1911a62a82e8c42c8: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/wal/wals/4378340ddf0942f6a28ee77b11b798c6
I20260926 11:33:08.647418 16823 ts_tablet_manager.cc:1939] T 4378340ddf0942f6a28ee77b11b798c6 P 734e79c295e541f1911a62a82e8c42c8: Deleting consensus metadata
W20260926 11:33:08.647809 16523 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:33:08.647405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43321 (local address 127.0.0.1:38538)
0926 11:33:08.647591 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:08.647670 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43321: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 11:33:08.647990 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 16521
W20260926 11:33:08.647996 16314 catalog_manager.cc:4584] TS 119a39a719be45f88987bfa5a856ace8 (127.0.98.1:43321): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4378340ddf0942f6a28ee77b11b798c6: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43321: connect: Connection reset by peer (error 104)
I20260926 11:33:08.650333 16314 catalog_manager.cc:4841] TS 734e79c295e541f1911a62a82e8c42c8 (127.0.98.3:42483): tablet 4378340ddf0942f6a28ee77b11b798c6 (table test-table [id=bc3ed7211ab249b986fc66ada717b1ab]) successfully deleted
I20260926 11:33:08.654057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 16659
I20260926 11:33:08.659994 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 16295
I20260926 11:33:28.666683 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44617
--webserver_interface=127.0.98.62
--webserver_port=40365
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:33:28.780272 16825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:28.780563 16825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:28.780640 16825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:28.784597 16825 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 11:33:28.784678 16825 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 11:33:28.784705 16825 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 11:33:28.784739 16825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:33:28.784776 16825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:28.784822 16825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:28.784868 16825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:33:28.784904 16825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:33:28.790176 16825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:28.791262 16825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:28.792310 16825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:33:28.800889 16833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:28.800992 16831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:28.801015 16830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:28.801074 16825 server_base.cc:1034] running on GCE node
I20260926 11:33:28.801656 16825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:28.802793 16825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:28.803965 16825 hybrid_clock.cc:648] HybridClock initialized: now 1790422408803948 us; error 29 us; skew 500 ppm
Sep 26 11:33:28 dist-test-slave-mrck krb5kdc[15749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:33:28.812655 16825 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:33:28.813676 16825 webserver.cc:460] Webserver started at http://127.0.98.62:40365/ using document root <none> and password file <none>
I20260926 11:33:28.814266 16825 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:28.814357 16825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:28.818269 16825 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:33:28.820495 16840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:28.821807 16825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:33:28.821946 16825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "4d9b63b3aa6645f49a00ad00664d0c09"
format_stamp: "Formatted at 2026-09-26 11:32:46 on dist-test-slave-mrck"
server_key: "dfe14d6a598c2a4601ddbf7f9c0ab01a"
server_key_iv: "617d94376fa144fc1b605ae895432b15"
server_key_version: "encryptionkey@0"
I20260926 11:33:28.822537 16825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:28.844388 16825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:28.847306 16825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:28.847504 16825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:28.854606 16825 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44617
I20260926 11:33:28.854880 16892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44617 every 8 connection(s)
I20260926 11:33:28.855677 16825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:33:28.860103 16893 sys_catalog.cc:263] Verifying existing consensus state
I20260926 11:33:28.861842 16893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09: Bootstrap starting.
I20260926 11:33:28.863210 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 16825
I20260926 11:33:28.863652 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43321
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44617
--webserver_port=39597
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:33:28.880203 16893 log.cc:824] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09: Log is configured to *not* fsync() on all Append() calls
I20260926 11:33:28.891587 16893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 11:33:28.892047 16893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09: Bootstrap complete.
I20260926 11:33:28.896329 16893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d9b63b3aa6645f49a00ad00664d0c09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44617 } }
I20260926 11:33:28.896811 16893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d9b63b3aa6645f49a00ad00664d0c09, State: Initialized, Role: FOLLOWER
I20260926 11:33:28.897318 16893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d9b63b3aa6645f49a00ad00664d0c09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44617 } }
I20260926 11:33:28.897464 16893 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:33:28.897549 16893 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:33:28.897660 16893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [term 2 FOLLOWER]: Advancing to term 3
I20260926 11:33:28.899580 16893 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d9b63b3aa6645f49a00ad00664d0c09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44617 } }
I20260926 11:33:28.899948 16893 leader_election.cc:302] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d9b63b3aa6645f49a00ad00664d0c09; no voters:
I20260926 11:33:28.900301 16893 leader_election.cc:288] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 11:33:28.900431 16897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 11:33:28.900652 16897 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [term 3 LEADER]: Becoming Leader. State: Replica: 4d9b63b3aa6645f49a00ad00664d0c09, State: Running, Role: LEADER
I20260926 11:33:28.901024 16897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d9b63b3aa6645f49a00ad00664d0c09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44617 } }
I20260926 11:33:28.901604 16893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:33:28.902371 16898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4d9b63b3aa6645f49a00ad00664d0c09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d9b63b3aa6645f49a00ad00664d0c09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44617 } } }
I20260926 11:33:28.902508 16898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [sys.catalog]: This master's current role is: LEADER
I20260926 11:33:28.903358 16906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:33:28.903512 16899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d9b63b3aa6645f49a00ad00664d0c09. Latest consensus state: current_term: 3 leader_uuid: "4d9b63b3aa6645f49a00ad00664d0c09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d9b63b3aa6645f49a00ad00664d0c09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44617 } } }
I20260926 11:33:28.903698 16899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09 [sys.catalog]: This master's current role is: LEADER
I20260926 11:33:28.908597 16906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:33:28.909559 16906 catalog_manager.cc:1259] Loaded cluster ID: 065689bab73e4a5db2358bc164e61a5c
I20260926 11:33:28.909626 16906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:33:28.912180 16906 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:33:28.912904 16906 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09: Loaded TSK: 0
I20260926 11:33:28.913002 16906 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4d9b63b3aa6645f49a00ad00664d0c09: Loaded TSK: 1
F20260926 11:33:28.913762 16906 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1ea94b79ad google::LogMessage::Fail() at ??:0
@ 0x7f1ea94ba88a google::LogMessage::SendToLog() at ??:0
@ 0x7f1ea94b74de google::LogMessage::Flush() at ??:0
@ 0x7f1ea94b8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1eac17b18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f1eac173f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f1eb3e1c7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f1eb3df699c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f1eb3df7178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f1eb3e38df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f1eb3e5782b std::function<>::operator()() at ??:0
@ 0x7f1eb3df6b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f1eb3df7f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1eb3df4579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f1eb3e38640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1eab7093d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1eab709c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f1eab70b3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f1eab6fa512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1eacd3e6db start_thread at ??:0
@ 0x7f1ea7fb561f clone at ??:0
W20260926 11:33:29.015144 16895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:29.015569 16895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:29.015638 16895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:29.021926 16895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:29.022042 16895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:29.022135 16895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:33:29.030753 16895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43321
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39597
--tserver_master_addrs=127.0.98.62:44617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:29.032334 16895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:29.033658 16895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:33:29.045228 16940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:29.045243 16941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:29.045660 16895 server_base.cc:1034] running on GCE node
W20260926 11:33:29.045722 16943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:29.046098 16895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:29.046801 16895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:29.047991 16895 hybrid_clock.cc:648] HybridClock initialized: now 1790422409047966 us; error 44 us; skew 500 ppm
Sep 26 11:33:29 dist-test-slave-mrck krb5kdc[15749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:33:29.051055 16895 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:33:29.052117 16895 webserver.cc:460] Webserver started at http://127.0.98.1:39597/ using document root <none> and password file <none>
I20260926 11:33:29.052724 16895 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:29.052819 16895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:29.057152 16895 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260926 11:33:29.059594 16950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:29.061101 16895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 11:33:29.061234 16895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "119a39a719be45f88987bfa5a856ace8"
format_stamp: "Formatted at 2026-09-26 11:32:46 on dist-test-slave-mrck"
server_key: "b0201dab8a58429f825ebe42dfa32cb1"
server_key_iv: "c091791fcf35001a00da506b2e31b5a3"
server_key_version: "encryptionkey@0"
I20260926 11:33:29.062014 16895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:29.087889 16895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:29.091261 16895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:29.091466 16895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:29.092095 16895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:33:29.093230 16957 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 11:33:29.098805 16895 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 11:33:29.098927 16895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260926 11:33:29.099013 16895 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 11:33:29.100826 16895 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 11:33:29.100893 16895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.001s
I20260926 11:33:29.101157 16957 tablet_bootstrap.cc:492] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8: Bootstrap starting.
I20260926 11:33:29.111605 16895 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43321
I20260926 11:33:29.112488 17064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43321 every 8 connection(s)
I20260926 11:33:29.112983 16895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:33:29.121320 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 16895
I20260926 11:33:29.121831 16957 log.cc:824] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8: Log is configured to *not* fsync() on all Append() calls
I20260926 11:33:29.121845 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43335
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44617
--webserver_port=46795
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:33:29.129243 16957 tablet_bootstrap.cc:492] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 11:33:29.129660 16957 tablet_bootstrap.cc:492] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8: Bootstrap complete.
I20260926 11:33:29.131047 16957 ts_tablet_manager.cc:1397] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8: Time spent bootstrapping tablet: real 0.030s user 0.019s sys 0.009s
I20260926 11:33:29.135233 16957 raft_consensus.cc:348] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734e79c295e541f1911a62a82e8c42c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42483 } } peers { permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 } } peers { permanent_uuid: "119a39a719be45f88987bfa5a856ace8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43321 } }
I20260926 11:33:29.135952 16957 raft_consensus.cc:729] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 119a39a719be45f88987bfa5a856ace8, State: Initialized, Role: FOLLOWER
I20260926 11:33:29.136554 16957 consensus_queue.cc:260] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734e79c295e541f1911a62a82e8c42c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42483 } } peers { permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 } } peers { permanent_uuid: "119a39a719be45f88987bfa5a856ace8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43321 } }
I20260926 11:33:29.137780 16957 ts_tablet_manager.cc:1428] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260926 11:33:29.158464 17067 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:33:29.115457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.1:47337)
0926 11:33:29.116075 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:29.116134 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:29.117182 (+ 1048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:29.157659 (+ 40477us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":176,"threads_started":1}
W20260926 11:33:29.158962 17065 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44617 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44617: Client connection negotiation failed: client connection to 127.0.98.62:44617: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 11:33:29.159721 17067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:29.159487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.1:40845)
0926 11:33:29.159605 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:29.159668 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 11:33:29.160410 17067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:29.160234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.1:37065)
0926 11:33:29.160324 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:29.160365 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 11:33:29.160625 17065 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 11:33:29.245810 17069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:29.246062 17069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:29.246112 17069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:29.250182 17069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:29.250243 17069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:29.250326 17069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:33:29.255523 17069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43335
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46795
--tserver_master_addrs=127.0.98.62:44617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:29.256614 17069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:29.257696 17069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:33:29.267536 17076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:29.267578 17069 server_base.cc:1034] running on GCE node
W20260926 11:33:29.267521 17075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:29.267529 17078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:29.268187 17069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:29.268851 17069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:29.270038 17069 hybrid_clock.cc:648] HybridClock initialized: now 1790422409270012 us; error 36 us; skew 500 ppm
Sep 26 11:33:29 dist-test-slave-mrck krb5kdc[15749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:33:29.272953 17069 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:33:29.273903 17069 webserver.cc:460] Webserver started at http://127.0.98.2:46795/ using document root <none> and password file <none>
I20260926 11:33:29.274478 17069 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:29.274580 17069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:29.278448 17069 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260926 11:33:29.280560 17085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:29.281796 17069 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:33:29.281939 17069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "9efbdbfb4d664f32a69fe60963885f21"
format_stamp: "Formatted at 2026-09-26 11:32:46 on dist-test-slave-mrck"
server_key: "a7085119b68a3212b78d9c5abdda1472"
server_key_iv: "dd5f76d2b089eb8e2225aaf35cda8115"
server_key_version: "encryptionkey@0"
I20260926 11:33:29.282481 17069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:29.314401 17069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:29.317741 17069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:29.317958 17069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:29.318604 17069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:33:29.319445 17069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:33:29.319512 17069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:29.319595 17069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:33:29.319643 17069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:29.329097 17069 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43335
I20260926 11:33:29.329116 17198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43335 every 8 connection(s)
I20260926 11:33:29.330196 17069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:33:29.333899 17201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:29.332365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.2:58967)
0926 11:33:29.333035 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:29.333151 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":113,"threads_started":1}
W20260926 11:33:29.334406 17199 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44617 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44617: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
I20260926 11:33:29.334926 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 17069
I20260926 11:33:29.335104 17201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:29.334862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.2:52369)
0926 11:33:29.334968 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:29.335041 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 11:33:29.335469 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42483
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44617
--webserver_port=36973
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:33:29.335945 17201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:29.335645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.2:49777)
0926 11:33:29.335835 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:29.335890 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260926 11:33:29.336174 17199 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 11:33:29.448096 17202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:29.448393 17202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:29.448472 17202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:29.452643 17202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:29.452728 17202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:29.452818 17202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:33:29.458371 17202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42483
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36973
--tserver_master_addrs=127.0.98.62:44617
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:29.459503 17202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:29.460600 17202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:33:29.470361 17207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:29.470343 17210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:29.470347 17208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:29.470824 17202 server_base.cc:1034] running on GCE node
I20260926 11:33:29.471320 17202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:29.472007 17202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:29.473191 17202 hybrid_clock.cc:648] HybridClock initialized: now 1790422409473173 us; error 33 us; skew 500 ppm
Sep 26 11:33:29 dist-test-slave-mrck krb5kdc[15749](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:33:29.475941 17202 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:33:29.476838 17202 webserver.cc:460] Webserver started at http://127.0.98.3:36973/ using document root <none> and password file <none>
I20260926 11:33:29.477411 17202 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:29.477499 17202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:29.481352 17202 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:33:29.483351 17217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:29.484591 17202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:33:29.484726 17202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "734e79c295e541f1911a62a82e8c42c8"
format_stamp: "Formatted at 2026-09-26 11:32:47 on dist-test-slave-mrck"
server_key: "c1e319c1370928ee0f125e9a8317fef0"
server_key_iv: "c0f9b0b6715c8979445be1ae5b83570e"
server_key_version: "encryptionkey@0"
I20260926 11:33:29.485277 17202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:29.505139 17202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:29.508555 17202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:29.508770 17202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:29.509382 17202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:33:29.510241 17202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:33:29.510311 17202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:29.510381 17202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:33:29.510427 17202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:29.519305 17202 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42483
I20260926 11:33:29.519325 17330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42483 every 8 connection(s)
I20260926 11:33:29.520355 17202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:33:29.522141 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 17202
I20260926 11:33:29.524319 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:29.522624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.3:47283)
0926 11:33:29.523413 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:29.523541 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":145,"threads_started":1}
W20260926 11:33:29.524724 17331 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44617 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:44617: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
I20260926 11:33:29.525288 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:29.525082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.3:48307)
0926 11:33:29.525189 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:29.525234 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 11:33:29.525863 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:29.525666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.3:33991)
0926 11:33:29.525764 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:29.525809 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 11:33:29.526036 17331 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 11:33:30.161568 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:30.161160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.1:50369)
0926 11:33:30.161399 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:30.161456 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":71,"threads_started":1}
I20260926 11:33:30.337100 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:30.336691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.2:58423)
0926 11:33:30.336924 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:30.336977 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":73,"threads_started":1}
I20260926 11:33:30.527148 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:30.526629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.3:55535)
0926 11:33:30.526955 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:30.527040 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":91,"threads_started":1}
I20260926 11:33:30.572854 17338 raft_consensus.cc:482] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:33:30.573058 17338 raft_consensus.cc:504] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734e79c295e541f1911a62a82e8c42c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42483 } } peers { permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 } } peers { permanent_uuid: "119a39a719be45f88987bfa5a856ace8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43321 } }
I20260926 11:33:30.574179 17338 leader_election.cc:288] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 734e79c295e541f1911a62a82e8c42c8 (127.0.98.3:42483), 9efbdbfb4d664f32a69fe60963885f21 (127.0.98.2:43335)
Sep 26 11:33:30 dist-test-slave-mrck krb5kdc[15749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 26 11:33:30 dist-test-slave-mrck krb5kdc[15749](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260926 11:33:30.588923 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:30.574436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42483 (local address 127.0.98.1:60145)
0926 11:33:30.574606 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:30.574623 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:30.574735 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:30.576969 (+ 2234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:30.576974 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:30.577356 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:33:30.577866 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:30.577881 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:30.579208 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:30.579216 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:30.580223 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:30.580233 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:30.580926 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:30.580929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:30.581155 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:30.581875 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:33:30.581899 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:33:30.583792 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:33:30.585976 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:33:30.585982 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:33:30.585998 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:33:30.586314 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:33:30.586706 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:33:30.586709 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:33:30.586711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:33:30.586888 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:33:30.587687 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:33:30.587695 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:33:30.588119 (+ 424us) client_negotiation.cc:769] Sending connection context
0926 11:33:30.588350 (+ 231us) client_negotiation.cc:240] Negotiation successful
0926 11:33:30.588722 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":167,"spinlock_wait_cycles":11008}
I20260926 11:33:30.588995 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:30.574710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60145 (local address 127.0.98.3:42483)
0926 11:33:30.575129 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:30.575148 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:30.575179 (+ 31us) server_negotiation.cc:407] Connection header received
0926 11:33:30.576058 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:30.576067 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:30.576548 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:30.576894 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:30.578098 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:30.579056 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:30.580341 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:30.580802 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:30.580973 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:30.583956 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:30.583962 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:30.583974 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:33:30.584002 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:30.585853 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:30.586439 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:30.586443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:30.586447 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:30.586576 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:30.587017 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:30.587021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:30.587023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:30.587557 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:30.587698 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:30.588524 (+ 826us) server_negotiation.cc:299] Negotiation successful
0926 11:33:30.588792 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":102,"threads_started":1}
I20260926 11:33:30.588994 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:30.574625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60345 (local address 127.0.98.2:43335)
0926 11:33:30.574995 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:30.575020 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:30.575082 (+ 62us) server_negotiation.cc:407] Connection header received
0926 11:33:30.576058 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:30.576067 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:30.576547 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:30.576894 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:30.578063 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:30.579058 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:30.580341 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:30.580800 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:30.580974 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:30.583964 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:30.583969 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:30.583989 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:33:30.584016 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:30.585940 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:30.586448 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:30.586454 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:30.586464 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:30.586608 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:30.587017 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:30.587021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:30.587023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:30.587557 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:30.587698 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:30.588489 (+ 791us) server_negotiation.cc:299] Negotiation successful
0926 11:33:30.588792 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":143,"threads_started":1}
I20260926 11:33:30.588922 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:30.574436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43335 (local address 127.0.98.1:60345)
0926 11:33:30.574893 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:30.574906 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:30.575009 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:30.577392 (+ 2383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:30.577395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:30.577550 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:33:30.577866 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:30.577881 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:30.579226 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:30.579229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:30.580223 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:30.580233 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:30.580926 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:30.580930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:30.581156 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:30.581869 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:33:30.581885 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:33:30.583400 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:33:30.586065 (+ 2665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:33:30.586071 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:33:30.586074 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:33:30.586343 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:33:30.586705 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:33:30.586709 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:33:30.586711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:33:30.586888 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:33:30.587687 (+ 799us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:33:30.587695 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:33:30.588119 (+ 424us) client_negotiation.cc:769] Sending connection context
0926 11:33:30.588350 (+ 231us) client_negotiation.cc:240] Negotiation successful
0926 11:33:30.588777 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":480,"spinlock_wait_cycles":15616,"thread_start_us":81,"threads_started":1}
W20260926 11:33:30.591432 16952 leader_election.cc:341] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9efbdbfb4d664f32a69fe60963885f21 (127.0.98.2:43335): Not found: Tablet not found: 4378340ddf0942f6a28ee77b11b798c6
W20260926 11:33:30.591578 16953 leader_election.cc:341] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 734e79c295e541f1911a62a82e8c42c8 (127.0.98.3:42483): Not found: Tablet not found: 4378340ddf0942f6a28ee77b11b798c6
I20260926 11:33:30.591689 16953 leader_election.cc:302] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 119a39a719be45f88987bfa5a856ace8; no voters: 734e79c295e541f1911a62a82e8c42c8, 9efbdbfb4d664f32a69fe60963885f21
I20260926 11:33:30.592150 17338 raft_consensus.cc:2726] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 11:33:31.162760 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:31.162322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.1:58273)
0926 11:33:31.162580 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:31.162646 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":75,"threads_started":1}
I20260926 11:33:31.338193 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:31.337796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.2:54267)
0926 11:33:31.338045 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:31.338102 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":73,"threads_started":1}
I20260926 11:33:31.528285 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:31.527881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.3:56433)
0926 11:33:31.528136 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:31.528196 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":93,"threads_started":1}
I20260926 11:33:32.164189 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:32.163708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.1:46103)
0926 11:33:32.164026 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:32.164090 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":78,"threads_started":1}
I20260926 11:33:32.339442 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:32.338972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.2:34635)
0926 11:33:32.339254 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:32.339327 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":85,"threads_started":1}
I20260926 11:33:32.384505 17347 raft_consensus.cc:482] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:33:32.384670 17347 raft_consensus.cc:504] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734e79c295e541f1911a62a82e8c42c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42483 } } peers { permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 } } peers { permanent_uuid: "119a39a719be45f88987bfa5a856ace8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43321 } }
I20260926 11:33:32.385299 17347 leader_election.cc:288] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 734e79c295e541f1911a62a82e8c42c8 (127.0.98.3:42483), 9efbdbfb4d664f32a69fe60963885f21 (127.0.98.2:43335)
W20260926 11:33:32.385888 16952 leader_election.cc:341] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9efbdbfb4d664f32a69fe60963885f21 (127.0.98.2:43335): Not found: Tablet not found: 4378340ddf0942f6a28ee77b11b798c6
W20260926 11:33:32.385986 16953 leader_election.cc:341] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 734e79c295e541f1911a62a82e8c42c8 (127.0.98.3:42483): Not found: Tablet not found: 4378340ddf0942f6a28ee77b11b798c6
I20260926 11:33:32.386087 16953 leader_election.cc:302] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 119a39a719be45f88987bfa5a856ace8; no voters: 734e79c295e541f1911a62a82e8c42c8, 9efbdbfb4d664f32a69fe60963885f21
I20260926 11:33:32.386238 17347 raft_consensus.cc:2726] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 11:33:32.529369 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:32.528932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.3:40133)
0926 11:33:32.529200 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:32.529259 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":76,"threads_started":1}
I20260926 11:33:33.165552 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:33.165094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.1:36327)
0926 11:33:33.165380 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:33.165448 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":76,"threads_started":1}
I20260926 11:33:33.340792 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:33.340312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.2:45045)
0926 11:33:33.340627 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:33.340694 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":93,"threads_started":1}
I20260926 11:33:33.530524 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:33.530106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.3:40941)
0926 11:33:33.530380 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:33.530433 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":90,"threads_started":1}
I20260926 11:33:34.167121 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:34.166567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.1:51355)
0926 11:33:34.166913 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:34.166995 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":121,"threads_started":1}
I20260926 11:33:34.342154 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:34.341715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.2:48075)
0926 11:33:34.341985 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:34.342049 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":94,"threads_started":1}
I20260926 11:33:34.350493 17354 raft_consensus.cc:482] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:33:34.350625 17354 raft_consensus.cc:504] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734e79c295e541f1911a62a82e8c42c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42483 } } peers { permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 } } peers { permanent_uuid: "119a39a719be45f88987bfa5a856ace8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43321 } }
I20260926 11:33:34.351039 17354 leader_election.cc:288] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 734e79c295e541f1911a62a82e8c42c8 (127.0.98.3:42483), 9efbdbfb4d664f32a69fe60963885f21 (127.0.98.2:43335)
W20260926 11:33:34.351797 16953 leader_election.cc:341] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 734e79c295e541f1911a62a82e8c42c8 (127.0.98.3:42483): Not found: Tablet not found: 4378340ddf0942f6a28ee77b11b798c6
W20260926 11:33:34.351941 16952 leader_election.cc:341] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9efbdbfb4d664f32a69fe60963885f21 (127.0.98.2:43335): Not found: Tablet not found: 4378340ddf0942f6a28ee77b11b798c6
I20260926 11:33:34.352035 16952 leader_election.cc:302] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 119a39a719be45f88987bfa5a856ace8; no voters: 734e79c295e541f1911a62a82e8c42c8, 9efbdbfb4d664f32a69fe60963885f21
I20260926 11:33:34.352190 17354 raft_consensus.cc:2726] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 11:33:34.531955 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:34.531396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.3:39509)
0926 11:33:34.531708 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:34.531841 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":107,"threads_started":1}
I20260926 11:33:35.168241 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:35.167840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.1:36605)
0926 11:33:35.168087 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:35.168144 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":72,"threads_started":1}
I20260926 11:33:35.343398 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:35.342965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.2:41297)
0926 11:33:35.343245 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:35.343300 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":76,"threads_started":1}
I20260926 11:33:35.533092 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:35.532700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.3:57525)
0926 11:33:35.532930 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:35.532983 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":73,"threads_started":1}
I20260926 11:33:36.169442 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:36.169025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.1:56067)
0926 11:33:36.169287 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:36.169347 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":86,"threads_started":1}
I20260926 11:33:36.344692 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:36.344247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.2:52893)
0926 11:33:36.344534 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:36.344592 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":76,"threads_started":1}
I20260926 11:33:36.534405 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:36.533901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.3:41765)
0926 11:33:36.534250 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:36.534313 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":89,"threads_started":1}
I20260926 11:33:37.170676 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:37.170242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.1:57635)
0926 11:33:37.170509 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:37.170581 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":80,"threads_started":1}
I20260926 11:33:37.223371 17363 raft_consensus.cc:482] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:33:37.223510 17363 raft_consensus.cc:504] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734e79c295e541f1911a62a82e8c42c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42483 } } peers { permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 } } peers { permanent_uuid: "119a39a719be45f88987bfa5a856ace8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43321 } }
I20260926 11:33:37.224005 17363 leader_election.cc:288] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 734e79c295e541f1911a62a82e8c42c8 (127.0.98.3:42483), 9efbdbfb4d664f32a69fe60963885f21 (127.0.98.2:43335)
W20260926 11:33:37.224789 16952 leader_election.cc:341] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9efbdbfb4d664f32a69fe60963885f21 (127.0.98.2:43335): Not found: Tablet not found: 4378340ddf0942f6a28ee77b11b798c6
W20260926 11:33:37.224941 16953 leader_election.cc:341] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 734e79c295e541f1911a62a82e8c42c8 (127.0.98.3:42483): Not found: Tablet not found: 4378340ddf0942f6a28ee77b11b798c6
I20260926 11:33:37.225016 16953 leader_election.cc:302] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 119a39a719be45f88987bfa5a856ace8; no voters: 734e79c295e541f1911a62a82e8c42c8, 9efbdbfb4d664f32a69fe60963885f21
I20260926 11:33:37.225160 17363 raft_consensus.cc:2726] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 11:33:37.345920 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:37.345476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.2:54783)
0926 11:33:37.345761 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:37.345821 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":91,"threads_started":1}
I20260926 11:33:37.535844 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:37.535281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.3:58873)
0926 11:33:37.535602 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:37.535673 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":90,"threads_started":1}
I20260926 11:33:38.171944 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:38.171420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.1:60609)
0926 11:33:38.171718 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:38.171844 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":92,"threads_started":1}
I20260926 11:33:38.347234 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:38.346777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.2:47795)
0926 11:33:38.347065 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:38.347133 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":79,"threads_started":1}
I20260926 11:33:38.537142 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:38.536681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.3:49891)
0926 11:33:38.536960 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:38.537046 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":98,"threads_started":1}
I20260926 11:33:39.173341 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:39.172875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.1:59581)
0926 11:33:39.173160 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:39.173227 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":80,"threads_started":1}
I20260926 11:33:39.348668 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:39.348210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.2:35711)
0926 11:33:39.348501 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:39.348569 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":75,"threads_started":1}
I20260926 11:33:39.538489 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:39.538022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.3:59727)
0926 11:33:39.538323 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:39.538390 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":92,"threads_started":1}
I20260926 11:33:40.174597 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:40.174153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.1:38863)
0926 11:33:40.174436 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:40.174496 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":77,"threads_started":1}
I20260926 11:33:40.349814 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:40.349406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.2:40975)
0926 11:33:40.349660 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:40.349718 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":72,"threads_started":1}
I20260926 11:33:40.539806 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:40.539363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.3:54823)
0926 11:33:40.539627 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:40.539685 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":86,"threads_started":1}
I20260926 11:33:41.175951 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:41.175476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.1:33743)
0926 11:33:41.175781 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:41.175854 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":79,"threads_started":1}
I20260926 11:33:41.351070 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:41.350652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.2:53917)
0926 11:33:41.350904 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:41.350966 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":77,"threads_started":1}
I20260926 11:33:41.541113 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:41.540638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.3:39275)
0926 11:33:41.540939 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:41.541008 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":112,"threads_started":1}
I20260926 11:33:41.937039 17378 raft_consensus.cc:482] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:33:41.937167 17378 raft_consensus.cc:504] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "734e79c295e541f1911a62a82e8c42c8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42483 } } peers { permanent_uuid: "9efbdbfb4d664f32a69fe60963885f21" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43335 } } peers { permanent_uuid: "119a39a719be45f88987bfa5a856ace8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43321 } }
I20260926 11:33:41.937678 17378 leader_election.cc:288] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 734e79c295e541f1911a62a82e8c42c8 (127.0.98.3:42483), 9efbdbfb4d664f32a69fe60963885f21 (127.0.98.2:43335)
W20260926 11:33:41.938597 16953 leader_election.cc:341] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 734e79c295e541f1911a62a82e8c42c8 (127.0.98.3:42483): Not found: Tablet not found: 4378340ddf0942f6a28ee77b11b798c6
W20260926 11:33:41.938706 16952 leader_election.cc:341] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9efbdbfb4d664f32a69fe60963885f21 (127.0.98.2:43335): Not found: Tablet not found: 4378340ddf0942f6a28ee77b11b798c6
I20260926 11:33:41.938771 16952 leader_election.cc:302] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 119a39a719be45f88987bfa5a856ace8; no voters: 734e79c295e541f1911a62a82e8c42c8, 9efbdbfb4d664f32a69fe60963885f21
I20260926 11:33:41.938910 17378 raft_consensus.cc:2726] T 4378340ddf0942f6a28ee77b11b798c6 P 119a39a719be45f88987bfa5a856ace8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 11:33:42.177174 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:42.176743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.1:34003)
0926 11:33:42.177030 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:42.177082 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":112,"threads_started":1}
I20260926 11:33:42.352507 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:42.352054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.2:55751)
0926 11:33:42.352313 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:42.352383 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":69,"threads_started":1}
I20260926 11:33:42.542253 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:42.541839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.3:51973)
0926 11:33:42.542101 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:42.542160 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":76,"threads_started":1}
I20260926 11:33:43.178295 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:43.177911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.1:36217)
0926 11:33:43.178143 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:43.178199 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":75,"threads_started":1}
I20260926 11:33:43.353736 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:43.353283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.2:57015)
0926 11:33:43.353576 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:43.353638 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":79,"threads_started":1}
I20260926 11:33:43.543710 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:43.543252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.3:42373)
0926 11:33:43.543555 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:43.543617 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":93,"threads_started":1}
I20260926 11:33:44.179720 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:44.179287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.1:52203)
0926 11:33:44.179562 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:44.179625 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":74,"threads_started":1}
I20260926 11:33:44.355225 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:44.354768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44617 (local address 127.0.98.2:41825)
0926 11:33:44.355069 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:44.355130 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:44617: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":80,"threads_started":1}
I20260926 11:33:44.530972 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 16895
I20260926 11:33:44.537652 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 17069
I20260926 11:33:44.542795 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 17202
2026-09-26T11:33:44Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59537 ms)
[----------] 30 tests from SecurityITest (244249 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:33:44 dist-test-slave-mrck krb5kdc[17389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:33:44 dist-test-slave-mrck krb5kdc[17389](info): set up 2 sockets
Sep 26 11:33:44 dist-test-slave-mrck krb5kdc[17389](info): commencing operation
krb5kdc: starting...
W20260926 11:33:45.595597 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:33:45 dist-test-slave-mrck krb5kdc[17389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:33:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:33:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:33:45.716138 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44095
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44095
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:33:45.826339 17402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:45.826622 17402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:45.826665 17402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:45.830588 17402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:33:45.830643 17402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:45.830662 17402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:45.830678 17402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:33:45.830693 17402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:33:45.835938 17402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:45.836956 17402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:45.837937 17402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 11:33:45.847404 17402 server_base.cc:1034] running on GCE node
W20260926 11:33:45.847371 17408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:45.847325 17407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:45.847342 17410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:45.848242 17402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:45.849313 17402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:45.850472 17402 hybrid_clock.cc:648] HybridClock initialized: now 1790422425850459 us; error 25 us; skew 500 ppm
Sep 26 11:33:45 dist-test-slave-mrck krb5kdc[17389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:33:45.856853 17402 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:33:45.857841 17402 webserver.cc:460] Webserver started at http://127.0.98.62:36327/ using document root <none> and password file <none>
I20260926 11:33:45.858431 17402 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:45.858500 17402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:45.858726 17402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:33:45.860888 17402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "9a22bef8fb8045d2991b585c9fbfa4fd"
format_stamp: "Formatted at 2026-09-26 11:33:45 on dist-test-slave-mrck"
server_key: "4ef22ca370a7c718167ec7fd7dcff0c8"
server_key_iv: "101720faff78b4c610d2ea37a4e718ae"
server_key_version: "encryptionkey@0"
I20260926 11:33:45.861497 17402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "9a22bef8fb8045d2991b585c9fbfa4fd"
format_stamp: "Formatted at 2026-09-26 11:33:45 on dist-test-slave-mrck"
server_key: "4ef22ca370a7c718167ec7fd7dcff0c8"
server_key_iv: "101720faff78b4c610d2ea37a4e718ae"
server_key_version: "encryptionkey@0"
I20260926 11:33:45.865216 17402 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:33:45.867669 17417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:45.868952 17402 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:33:45.869097 17402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "9a22bef8fb8045d2991b585c9fbfa4fd"
format_stamp: "Formatted at 2026-09-26 11:33:45 on dist-test-slave-mrck"
server_key: "4ef22ca370a7c718167ec7fd7dcff0c8"
server_key_iv: "101720faff78b4c610d2ea37a4e718ae"
server_key_version: "encryptionkey@0"
I20260926 11:33:45.869218 17402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:45.883621 17402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:45.887060 17402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:45.887274 17402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:45.895190 17402 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44095
I20260926 11:33:45.895208 17469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44095 every 8 connection(s)
I20260926 11:33:45.896219 17402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:33:45.899155 17470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:33:45.902293 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 17402
I20260926 11:33:45.902426 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:33:45.902697 392 external_mini_cluster.cc:1428] Setting key 64d806895a8ded323c54edd757e5dae2
I20260926 11:33:45.905449 17470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a22bef8fb8045d2991b585c9fbfa4fd: Bootstrap starting.
I20260926 11:33:45.908072 17470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a22bef8fb8045d2991b585c9fbfa4fd: Neither blocks nor log segments found. Creating new log.
I20260926 11:33:45.908972 17470 log.cc:824] T 00000000000000000000000000000000 P 9a22bef8fb8045d2991b585c9fbfa4fd: Log is configured to *not* fsync() on all Append() calls
I20260926 11:33:45.911212 17470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a22bef8fb8045d2991b585c9fbfa4fd: No bootstrap required, opened a new log
Sep 26 11:33:45 dist-test-slave-mrck krb5kdc[17389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:33:45.914338 17470 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a22bef8fb8045d2991b585c9fbfa4fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a22bef8fb8045d2991b585c9fbfa4fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44095 } }
I20260926 11:33:45.914561 17470 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a22bef8fb8045d2991b585c9fbfa4fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:33:45.914623 17470 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a22bef8fb8045d2991b585c9fbfa4fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a22bef8fb8045d2991b585c9fbfa4fd, State: Initialized, Role: FOLLOWER
I20260926 11:33:45.915035 17470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a22bef8fb8045d2991b585c9fbfa4fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a22bef8fb8045d2991b585c9fbfa4fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44095 } }
I20260926 11:33:45.915184 17470 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a22bef8fb8045d2991b585c9fbfa4fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:33:45.915272 17470 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a22bef8fb8045d2991b585c9fbfa4fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:33:45.915390 17470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a22bef8fb8045d2991b585c9fbfa4fd [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:33:45.916337 17470 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a22bef8fb8045d2991b585c9fbfa4fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a22bef8fb8045d2991b585c9fbfa4fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44095 } }
I20260926 11:33:45.916690 17470 leader_election.cc:302] T 00000000000000000000000000000000 P 9a22bef8fb8045d2991b585c9fbfa4fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a22bef8fb8045d2991b585c9fbfa4fd; no voters:
I20260926 11:33:45.917071 17470 leader_election.cc:288] T 00000000000000000000000000000000 P 9a22bef8fb8045d2991b585c9fbfa4fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:33:45.917167 17475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a22bef8fb8045d2991b585c9fbfa4fd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:33:45.917397 17475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a22bef8fb8045d2991b585c9fbfa4fd [term 1 LEADER]: Becoming Leader. State: Replica: 9a22bef8fb8045d2991b585c9fbfa4fd, State: Running, Role: LEADER
I20260926 11:33:45.917757 17475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a22bef8fb8045d2991b585c9fbfa4fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a22bef8fb8045d2991b585c9fbfa4fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44095 } }
I20260926 11:33:45.918143 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:45.904209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34682 (local address 127.0.98.62:44095)
0926 11:33:45.904634 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:45.904658 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:45.904693 (+ 35us) server_negotiation.cc:407] Connection header received
0926 11:33:45.905315 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:45.905332 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:45.905713 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:45.906000 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:45.906899 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:45.907948 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:45.909124 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:45.909787 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:45.909904 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:45.912628 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:45.912633 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:45.912646 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:33:45.912671 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:45.915043 (+ 2372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:45.915533 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:45.915541 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:45.915550 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:45.915658 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:45.915957 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:45.915961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:45.915963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:45.916343 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:45.916570 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:45.916798 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 11:33:45.917125 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":154,"threads_started":1}
I20260926 11:33:45.918479 17470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a22bef8fb8045d2991b585c9fbfa4fd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:33:45.918936 17475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a22bef8fb8045d2991b585c9fbfa4fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a22bef8fb8045d2991b585c9fbfa4fd. Latest consensus state: current_term: 1 leader_uuid: "9a22bef8fb8045d2991b585c9fbfa4fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a22bef8fb8045d2991b585c9fbfa4fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44095 } } }
I20260926 11:33:45.919135 17475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a22bef8fb8045d2991b585c9fbfa4fd [sys.catalog]: This master's current role is: LEADER
I20260926 11:33:45.919373 17476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a22bef8fb8045d2991b585c9fbfa4fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a22bef8fb8045d2991b585c9fbfa4fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a22bef8fb8045d2991b585c9fbfa4fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44095 } } }
I20260926 11:33:45.919550 17476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a22bef8fb8045d2991b585c9fbfa4fd [sys.catalog]: This master's current role is: LEADER
I20260926 11:33:45.920367 17479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:33:45.923213 17479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:33:45.928905 17479 catalog_manager.cc:1347] Generated new cluster ID: a51227959cf749f592996491ead9f166
I20260926 11:33:45.929020 17479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:33:45.944300 17479 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:33:45.945279 17479 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:33:45.955291 17479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a22bef8fb8045d2991b585c9fbfa4fd: Generated new TSK 0
I20260926 11:33:45.956126 17479 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:33:45.984572 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44095
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:33:46.097760 17496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:46.098031 17496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:46.098075 17496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:46.102509 17496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:46.102569 17496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:46.102666 17496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:33:46.108162 17496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:46.109305 17496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:46.110365 17496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:33:46.119887 17501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:46.119992 17502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:46.119887 17504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:46.120375 17496 server_base.cc:1034] running on GCE node
I20260926 11:33:46.120867 17496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:46.121533 17496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:46.122712 17496 hybrid_clock.cc:648] HybridClock initialized: now 1790422426122701 us; error 39 us; skew 500 ppm
Sep 26 11:33:46 dist-test-slave-mrck krb5kdc[17389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:33:46.125516 17496 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:33:46.126468 17496 webserver.cc:460] Webserver started at http://127.0.98.1:43025/ using document root <none> and password file <none>
I20260926 11:33:46.127044 17496 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:46.127135 17496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:46.127362 17496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:33:46.129417 17496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-0/data/instance:
uuid: "66114ce880744f0fb8e104a23344c94e"
format_stamp: "Formatted at 2026-09-26 11:33:46 on dist-test-slave-mrck"
server_key: "2d5def59d1fecd4c1e9014c480156041"
server_key_iv: "479379d84259e59b1314c6648e19af95"
server_key_version: "encryptionkey@0"
I20260926 11:33:46.129972 17496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "66114ce880744f0fb8e104a23344c94e"
format_stamp: "Formatted at 2026-09-26 11:33:46 on dist-test-slave-mrck"
server_key: "2d5def59d1fecd4c1e9014c480156041"
server_key_iv: "479379d84259e59b1314c6648e19af95"
server_key_version: "encryptionkey@0"
I20260926 11:33:46.133633 17496 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260926 11:33:46.136021 17511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:46.137249 17496 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:33:46.137404 17496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "66114ce880744f0fb8e104a23344c94e"
format_stamp: "Formatted at 2026-09-26 11:33:46 on dist-test-slave-mrck"
server_key: "2d5def59d1fecd4c1e9014c480156041"
server_key_iv: "479379d84259e59b1314c6648e19af95"
server_key_version: "encryptionkey@0"
I20260926 11:33:46.137529 17496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:46.161720 17496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:46.165122 17496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:46.165334 17496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:46.166108 17496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:33:46.167122 17496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:33:46.167212 17496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:46.167291 17496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:33:46.167346 17496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:46.177217 17496 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39819
I20260926 11:33:46.177246 17624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39819 every 8 connection(s)
I20260926 11:33:46.178179 17496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:33:46.181267 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 17496
I20260926 11:33:46.181413 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:33:46.181732 392 external_mini_cluster.cc:1428] Setting key 0777c573fbd4e76634ba3eeeaa3f4a6b
Sep 26 11:33:46 dist-test-slave-mrck krb5kdc[17389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 11:33:46.192203 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:46.180228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46773 (local address 127.0.98.62:44095)
0926 11:33:46.180396 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:46.180400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:46.181527 (+ 1127us) server_negotiation.cc:407] Connection header received
0926 11:33:46.182551 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:46.182554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:46.182597 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:46.182672 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:46.184047 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.184571 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.185690 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.185942 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.186050 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:46.188589 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:46.188593 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:46.188595 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:33:46.188618 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:46.190088 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:46.190625 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:46.190630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:46.190632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:46.190714 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:46.191051 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:46.191056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:46.191058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:46.191234 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:46.191337 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:46.191892 (+ 555us) server_negotiation.cc:299] Negotiation successful
0926 11:33:46.192026 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 11:33:46.193161 17627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:46.180645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44095 (local address 127.0.98.1:46773)
0926 11:33:46.181360 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:46.181420 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:46.182344 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:46.182839 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:46.182848 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:46.183362 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:33:46.183897 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:46.183921 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.184695 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.184701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:46.185568 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:46.185577 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.186086 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.186090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:46.186255 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:46.186887 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:33:46.186902 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:33:46.188424 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:33:46.190205 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:33:46.190211 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:33:46.190227 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:33:46.190521 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:33:46.190804 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:33:46.190809 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:33:46.190811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:33:46.190956 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:33:46.191383 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:33:46.191390 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:33:46.191631 (+ 241us) client_negotiation.cc:769] Sending connection context
0926 11:33:46.191841 (+ 210us) client_negotiation.cc:240] Negotiation successful
0926 11:33:46.192198 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":119,"threads_started":1}
I20260926 11:33:46.194490 17625 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44095
I20260926 11:33:46.194754 17625 heartbeater.cc:464] Registering TS with master...
I20260926 11:33:46.195353 17625 heartbeater.cc:511] Master 127.0.98.62:44095 requested a full tablet report, sending...
I20260926 11:33:46.196986 17428 ts_manager.cc:194] Registered new tserver with Master: 66114ce880744f0fb8e104a23344c94e (127.0.98.1:39819)
I20260926 11:33:46.198273 17428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46773
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:33:46.210412 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44095
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:33:46.323988 17630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:46.324281 17630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:46.324327 17630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:46.328527 17630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:46.328596 17630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:46.328660 17630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:33:46.333940 17630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:46.334992 17630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:46.336098 17630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:33:46.345995 17638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:46.345985 17636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:46.346166 17630 server_base.cc:1034] running on GCE node
W20260926 11:33:46.346066 17635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:46.346719 17630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:46.347404 17630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:46.348573 17630 hybrid_clock.cc:648] HybridClock initialized: now 1790422426348557 us; error 28 us; skew 500 ppm
Sep 26 11:33:46 dist-test-slave-mrck krb5kdc[17389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:33:46.351394 17630 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:33:46.352383 17630 webserver.cc:460] Webserver started at http://127.0.98.2:35957/ using document root <none> and password file <none>
I20260926 11:33:46.352965 17630 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:46.353055 17630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:46.353292 17630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:33:46.355361 17630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-1/data/instance:
uuid: "325431e086e7403abdf19a759cd4c809"
format_stamp: "Formatted at 2026-09-26 11:33:46 on dist-test-slave-mrck"
server_key: "f13111ee2dacf21217bfbc64f1c105a5"
server_key_iv: "0d09e5c1e6591d57473a18c56a48e0ad"
server_key_version: "encryptionkey@0"
I20260926 11:33:46.355981 17630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "325431e086e7403abdf19a759cd4c809"
format_stamp: "Formatted at 2026-09-26 11:33:46 on dist-test-slave-mrck"
server_key: "f13111ee2dacf21217bfbc64f1c105a5"
server_key_iv: "0d09e5c1e6591d57473a18c56a48e0ad"
server_key_version: "encryptionkey@0"
I20260926 11:33:46.359804 17630 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:33:46.362214 17645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:46.363447 17630 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:33:46.363595 17630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "325431e086e7403abdf19a759cd4c809"
format_stamp: "Formatted at 2026-09-26 11:33:46 on dist-test-slave-mrck"
server_key: "f13111ee2dacf21217bfbc64f1c105a5"
server_key_iv: "0d09e5c1e6591d57473a18c56a48e0ad"
server_key_version: "encryptionkey@0"
I20260926 11:33:46.363708 17630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:46.387076 17630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:46.390511 17630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:46.390724 17630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:46.391377 17630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:33:46.392282 17630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:33:46.392352 17630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:46.392421 17630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:33:46.392470 17630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:46.401521 17630 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40433
I20260926 11:33:46.401539 17758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40433 every 8 connection(s)
I20260926 11:33:46.402439 17630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:33:46.407141 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 17630
I20260926 11:33:46.407284 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:33:46.407544 392 external_mini_cluster.cc:1428] Setting key db1b3bc40786d8383d95964edbeb2f8f
Sep 26 11:33:46 dist-test-slave-mrck krb5kdc[17389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 11:33:46.417636 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:46.404364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35131 (local address 127.0.98.62:44095)
0926 11:33:46.404505 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:46.404509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:46.405363 (+ 854us) server_negotiation.cc:407] Connection header received
0926 11:33:46.406291 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:46.406294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:46.406351 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:46.406477 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:46.407865 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.408627 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.410448 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.410757 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.410880 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:46.413824 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:46.413827 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:46.413829 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:33:46.413852 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:46.415268 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:46.415931 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:46.415933 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:46.415935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:46.416007 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:46.416401 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:46.416403 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:46.416404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:46.416575 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:46.416646 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:46.417350 (+ 704us) server_negotiation.cc:299] Negotiation successful
0926 11:33:46.417489 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 11:33:46.418589 17761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:46.404757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44095 (local address 127.0.98.2:35131)
0926 11:33:46.405205 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:46.405248 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:46.406080 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:46.406667 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:46.406674 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:46.407060 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:33:46.407679 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:46.407695 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.408804 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.408810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:46.410313 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:46.410328 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.410915 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.410918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:46.411074 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:46.411698 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:33:46.411715 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:33:46.413646 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:33:46.415390 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:33:46.415396 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:33:46.415413 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:33:46.415831 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:33:46.416104 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:33:46.416110 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:33:46.416115 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:33:46.416302 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:33:46.416680 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:33:46.416686 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:33:46.417044 (+ 358us) client_negotiation.cc:769] Sending connection context
0926 11:33:46.417255 (+ 211us) client_negotiation.cc:240] Negotiation successful
0926 11:33:46.417642 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":115,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 11:33:46.419950 17759 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44095
I20260926 11:33:46.420218 17759 heartbeater.cc:464] Registering TS with master...
I20260926 11:33:46.420791 17759 heartbeater.cc:511] Master 127.0.98.62:44095 requested a full tablet report, sending...
I20260926 11:33:46.421921 17428 ts_manager.cc:194] Registered new tserver with Master: 325431e086e7403abdf19a759cd4c809 (127.0.98.2:40433)
I20260926 11:33:46.422514 17428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:33:46.435398 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44095
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:33:46.547676 17764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:46.547974 17764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:46.548019 17764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:46.552227 17764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:46.552301 17764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:46.552363 17764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:33:46.557849 17764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:46.558916 17764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:46.559994 17764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:33:46.569904 17770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:46.569944 17772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:46.569906 17769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:46.570202 17764 server_base.cc:1034] running on GCE node
I20260926 11:33:46.570700 17764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:46.571408 17764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:46.572582 17764 hybrid_clock.cc:648] HybridClock initialized: now 1790422426572544 us; error 51 us; skew 500 ppm
Sep 26 11:33:46 dist-test-slave-mrck krb5kdc[17389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:33:46.575413 17764 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:33:46.576375 17764 webserver.cc:460] Webserver started at http://127.0.98.3:38515/ using document root <none> and password file <none>
I20260926 11:33:46.576953 17764 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:46.577044 17764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:46.577271 17764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:33:46.579298 17764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-2/data/instance:
uuid: "c893fecb8c0b4bdaa6226426c4200c80"
format_stamp: "Formatted at 2026-09-26 11:33:46 on dist-test-slave-mrck"
server_key: "aba625d24e07dccb34643d85ea432e00"
server_key_iv: "3e497794a7287a2a221ae4d1ff42cab0"
server_key_version: "encryptionkey@0"
I20260926 11:33:46.579952 17764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c893fecb8c0b4bdaa6226426c4200c80"
format_stamp: "Formatted at 2026-09-26 11:33:46 on dist-test-slave-mrck"
server_key: "aba625d24e07dccb34643d85ea432e00"
server_key_iv: "3e497794a7287a2a221ae4d1ff42cab0"
server_key_version: "encryptionkey@0"
I20260926 11:33:46.583822 17764 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 11:33:46.586308 17779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:46.587633 17764 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 11:33:46.587819 17764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "c893fecb8c0b4bdaa6226426c4200c80"
format_stamp: "Formatted at 2026-09-26 11:33:46 on dist-test-slave-mrck"
server_key: "aba625d24e07dccb34643d85ea432e00"
server_key_iv: "3e497794a7287a2a221ae4d1ff42cab0"
server_key_version: "encryptionkey@0"
I20260926 11:33:46.587955 17764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:46.618999 17764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:46.622967 17764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:46.623216 17764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:46.623951 17764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:33:46.624850 17764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:33:46.624897 17764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:46.624938 17764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:33:46.624959 17764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:46.633908 17764 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45201
I20260926 11:33:46.633932 17892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45201 every 8 connection(s)
I20260926 11:33:46.634946 17764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:33:46.642403 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 17764
I20260926 11:33:46.642546 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:33:46.642863 392 external_mini_cluster.cc:1428] Setting key 818c0ff8642df6e11e4e17afc069042a
Sep 26 11:33:46 dist-test-slave-mrck krb5kdc[17389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422426, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:33:46.649296 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:46.636901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35581 (local address 127.0.98.62:44095)
0926 11:33:46.637073 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:46.637077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:46.637936 (+ 859us) server_negotiation.cc:407] Connection header received
0926 11:33:46.639250 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:46.639253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:46.639307 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:46.639414 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:46.640698 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.641308 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.642588 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.642917 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.643038 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:46.646011 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:46.646014 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:46.646017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:33:46.646041 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:46.647358 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:46.647922 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:46.647926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:46.647928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:46.648011 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:46.648331 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:46.648333 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:46.648335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:46.648527 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:46.648602 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:46.649069 (+ 467us) server_negotiation.cc:299] Negotiation successful
0926 11:33:46.649176 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 11:33:46.650228 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:46.637314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44095 (local address 127.0.98.3:35581)
0926 11:33:46.637774 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:46.637819 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:46.639032 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:46.639599 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:46.639606 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:46.640018 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:33:46.640557 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:46.640572 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.641432 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.641437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:46.642424 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:46.642433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.643079 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.643082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:46.643230 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:46.644149 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:33:46.644166 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:33:46.645856 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:33:46.647470 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:33:46.647475 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:33:46.647488 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:33:46.647800 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:33:46.648108 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:33:46.648110 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:33:46.648111 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:33:46.648244 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:33:46.648611 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:33:46.648615 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:33:46.648855 (+ 240us) client_negotiation.cc:769] Sending connection context
0926 11:33:46.649006 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 11:33:46.649274 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":121,"threads_started":1}
I20260926 11:33:46.651439 17893 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44095
I20260926 11:33:46.651693 17893 heartbeater.cc:464] Registering TS with master...
I20260926 11:33:46.652222 17893 heartbeater.cc:511] Master 127.0.98.62:44095 requested a full tablet report, sending...
I20260926 11:33:46.653364 17428 ts_manager.cc:194] Registered new tserver with Master: c893fecb8c0b4bdaa6226426c4200c80 (127.0.98.3:45201)
I20260926 11:33:46.653959 17428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35581
I20260926 11:33:46.656819 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:33:46.666021 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:46.658706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34690 (local address 127.0.98.62:44095)
0926 11:33:46.658836 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:46.658842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:46.659040 (+ 198us) server_negotiation.cc:407] Connection header received
0926 11:33:46.659190 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:46.659193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:46.659235 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:46.659326 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:46.660105 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.660631 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.661468 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.661704 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.661813 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:46.663142 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:46.663146 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:46.663149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:33:46.663174 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:46.664363 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:46.664772 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:46.664777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:46.664779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:46.664880 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:46.665150 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:46.665154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:46.665156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:46.665375 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:46.665525 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:46.665740 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 11:33:46.665855 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 11:33:46.668790 17428 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34690:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:33:46.672070 17428 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:33:46.686581 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:46.681274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40433 (local address 127.0.0.1:49700)
0926 11:33:46.681507 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:46.681522 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:46.681613 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:46.682413 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:46.682416 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:46.682431 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:46.682552 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:46.682562 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.683782 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.683785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:46.684887 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:46.684898 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.685984 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.685991 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:46.686098 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:46.686134 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 11:33:46.686252 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 11:33:46.686402 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":6,"thread_start_us":88,"threads_started":1}
I20260926 11:33:46.686740 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:46.681619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49700 (local address 127.0.98.2:40433)
0926 11:33:46.681966 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:46.681979 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:46.682001 (+ 22us) server_negotiation.cc:407] Connection header received
0926 11:33:46.682079 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:46.682083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:46.682262 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:46.682690 (+ 428us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:46.682841 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.683639 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.685025 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.685842 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.685962 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:46.686122 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:46.686444 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 11:33:46.686557 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":140,"threads_started":1}
I20260926 11:33:46.686784 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:46.680576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45201 (local address 127.0.0.1:50696)
0926 11:33:46.681335 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:46.681348 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:46.681421 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:46.682795 (+ 1374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:46.682798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:46.682817 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:46.682898 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:46.682904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.684270 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.684274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:46.685339 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:46.685346 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.686345 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.686348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:46.686464 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:46.686484 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 11:33:46.686552 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 11:33:46.686638 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"mutex_wait_us":4,"thread_start_us":85,"threads_started":1}
I20260926 11:33:46.686857 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:46.680518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39819 (local address 127.0.0.1:49388)
0926 11:33:46.681115 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:46.681164 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:46.681315 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:46.682136 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:46.682142 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:46.682169 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:46.682287 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:46.682292 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.683618 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.683621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:46.685200 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:46.685208 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.686205 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.686209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:46.686336 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:46.686531 (+ 195us) client_negotiation.cc:769] Sending connection context
0926 11:33:46.686617 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 11:33:46.686693 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":214,"thread_start_us":87,"threads_started":1}
I20260926 11:33:46.687151 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:46.680632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49388 (local address 127.0.98.1:39819)
0926 11:33:46.680915 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:46.680936 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:46.681472 (+ 536us) server_negotiation.cc:407] Connection header received
0926 11:33:46.681559 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:46.681567 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:46.681785 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:46.681933 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:46.682461 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.683480 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.685327 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.685943 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.686054 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:46.686194 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:46.686857 (+ 663us) server_negotiation.cc:299] Negotiation successful
0926 11:33:46.686970 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":80,"threads_started":1}
I20260926 11:33:46.687223 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:46.681575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50696 (local address 127.0.98.3:45201)
0926 11:33:46.681966 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:46.681991 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:46.682009 (+ 18us) server_negotiation.cc:407] Connection header received
0926 11:33:46.682078 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:46.682083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:46.682487 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:46.682633 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:46.683033 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.684131 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.685447 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.686219 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.686335 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:46.686479 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:46.686932 (+ 453us) server_negotiation.cc:299] Negotiation successful
0926 11:33:46.687057 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":92,"threads_started":1}
I20260926 11:33:46.689808 17559 tablet_service.cc:1467] Processing CreateTablet for tablet a8299da1318f466aacf27c5a9ad25e6c (DEFAULT_TABLE table=test-table [id=7185b78dcfca4988b26262ce7ebed527]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:33:46.689934 17827 tablet_service.cc:1467] Processing CreateTablet for tablet a8299da1318f466aacf27c5a9ad25e6c (DEFAULT_TABLE table=test-table [id=7185b78dcfca4988b26262ce7ebed527]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:33:46.689934 17692 tablet_service.cc:1467] Processing CreateTablet for tablet a8299da1318f466aacf27c5a9ad25e6c (DEFAULT_TABLE table=test-table [id=7185b78dcfca4988b26262ce7ebed527]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:33:46.690917 17692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8299da1318f466aacf27c5a9ad25e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:33:46.690917 17827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8299da1318f466aacf27c5a9ad25e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:33:46.690919 17559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8299da1318f466aacf27c5a9ad25e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:33:46.697404 17910 tablet_bootstrap.cc:492] T a8299da1318f466aacf27c5a9ad25e6c P 66114ce880744f0fb8e104a23344c94e: Bootstrap starting.
I20260926 11:33:46.698663 17912 tablet_bootstrap.cc:492] T a8299da1318f466aacf27c5a9ad25e6c P 325431e086e7403abdf19a759cd4c809: Bootstrap starting.
I20260926 11:33:46.698879 17911 tablet_bootstrap.cc:492] T a8299da1318f466aacf27c5a9ad25e6c P c893fecb8c0b4bdaa6226426c4200c80: Bootstrap starting.
I20260926 11:33:46.699915 17910 tablet_bootstrap.cc:654] T a8299da1318f466aacf27c5a9ad25e6c P 66114ce880744f0fb8e104a23344c94e: Neither blocks nor log segments found. Creating new log.
I20260926 11:33:46.700799 17910 log.cc:824] T a8299da1318f466aacf27c5a9ad25e6c P 66114ce880744f0fb8e104a23344c94e: Log is configured to *not* fsync() on all Append() calls
I20260926 11:33:46.700822 17912 tablet_bootstrap.cc:654] T a8299da1318f466aacf27c5a9ad25e6c P 325431e086e7403abdf19a759cd4c809: Neither blocks nor log segments found. Creating new log.
I20260926 11:33:46.700822 17911 tablet_bootstrap.cc:654] T a8299da1318f466aacf27c5a9ad25e6c P c893fecb8c0b4bdaa6226426c4200c80: Neither blocks nor log segments found. Creating new log.
I20260926 11:33:46.702080 17912 log.cc:824] T a8299da1318f466aacf27c5a9ad25e6c P 325431e086e7403abdf19a759cd4c809: Log is configured to *not* fsync() on all Append() calls
I20260926 11:33:46.702078 17911 log.cc:824] T a8299da1318f466aacf27c5a9ad25e6c P c893fecb8c0b4bdaa6226426c4200c80: Log is configured to *not* fsync() on all Append() calls
I20260926 11:33:46.702750 17910 tablet_bootstrap.cc:492] T a8299da1318f466aacf27c5a9ad25e6c P 66114ce880744f0fb8e104a23344c94e: No bootstrap required, opened a new log
I20260926 11:33:46.702971 17910 ts_tablet_manager.cc:1397] T a8299da1318f466aacf27c5a9ad25e6c P 66114ce880744f0fb8e104a23344c94e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260926 11:33:46.704167 17912 tablet_bootstrap.cc:492] T a8299da1318f466aacf27c5a9ad25e6c P 325431e086e7403abdf19a759cd4c809: No bootstrap required, opened a new log
I20260926 11:33:46.704363 17912 ts_tablet_manager.cc:1397] T a8299da1318f466aacf27c5a9ad25e6c P 325431e086e7403abdf19a759cd4c809: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 11:33:46.704416 17911 tablet_bootstrap.cc:492] T a8299da1318f466aacf27c5a9ad25e6c P c893fecb8c0b4bdaa6226426c4200c80: No bootstrap required, opened a new log
I20260926 11:33:46.704572 17911 ts_tablet_manager.cc:1397] T a8299da1318f466aacf27c5a9ad25e6c P c893fecb8c0b4bdaa6226426c4200c80: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 11:33:46.707161 17910 raft_consensus.cc:348] T a8299da1318f466aacf27c5a9ad25e6c P 66114ce880744f0fb8e104a23344c94e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66114ce880744f0fb8e104a23344c94e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39819 } } peers { permanent_uuid: "c893fecb8c0b4bdaa6226426c4200c80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45201 } } peers { permanent_uuid: "325431e086e7403abdf19a759cd4c809" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40433 } }
I20260926 11:33:46.707428 17910 raft_consensus.cc:374] T a8299da1318f466aacf27c5a9ad25e6c P 66114ce880744f0fb8e104a23344c94e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:33:46.707500 17910 raft_consensus.cc:729] T a8299da1318f466aacf27c5a9ad25e6c P 66114ce880744f0fb8e104a23344c94e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66114ce880744f0fb8e104a23344c94e, State: Initialized, Role: FOLLOWER
I20260926 11:33:46.707994 17910 consensus_queue.cc:260] T a8299da1318f466aacf27c5a9ad25e6c P 66114ce880744f0fb8e104a23344c94e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66114ce880744f0fb8e104a23344c94e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39819 } } peers { permanent_uuid: "c893fecb8c0b4bdaa6226426c4200c80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45201 } } peers { permanent_uuid: "325431e086e7403abdf19a759cd4c809" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40433 } }
I20260926 11:33:46.708298 17911 raft_consensus.cc:348] T a8299da1318f466aacf27c5a9ad25e6c P c893fecb8c0b4bdaa6226426c4200c80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66114ce880744f0fb8e104a23344c94e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39819 } } peers { permanent_uuid: "c893fecb8c0b4bdaa6226426c4200c80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45201 } } peers { permanent_uuid: "325431e086e7403abdf19a759cd4c809" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40433 } }
I20260926 11:33:46.708298 17912 raft_consensus.cc:348] T a8299da1318f466aacf27c5a9ad25e6c P 325431e086e7403abdf19a759cd4c809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66114ce880744f0fb8e104a23344c94e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39819 } } peers { permanent_uuid: "c893fecb8c0b4bdaa6226426c4200c80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45201 } } peers { permanent_uuid: "325431e086e7403abdf19a759cd4c809" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40433 } }
I20260926 11:33:46.708563 17912 raft_consensus.cc:374] T a8299da1318f466aacf27c5a9ad25e6c P 325431e086e7403abdf19a759cd4c809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:33:46.708627 17911 raft_consensus.cc:374] T a8299da1318f466aacf27c5a9ad25e6c P c893fecb8c0b4bdaa6226426c4200c80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:33:46.708659 17625 heartbeater.cc:503] Master 127.0.98.62:44095 was elected leader, sending a full tablet report...
I20260926 11:33:46.708668 17912 raft_consensus.cc:729] T a8299da1318f466aacf27c5a9ad25e6c P 325431e086e7403abdf19a759cd4c809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 325431e086e7403abdf19a759cd4c809, State: Initialized, Role: FOLLOWER
I20260926 11:33:46.708774 17911 raft_consensus.cc:729] T a8299da1318f466aacf27c5a9ad25e6c P c893fecb8c0b4bdaa6226426c4200c80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c893fecb8c0b4bdaa6226426c4200c80, State: Initialized, Role: FOLLOWER
I20260926 11:33:46.709053 17910 ts_tablet_manager.cc:1428] T a8299da1318f466aacf27c5a9ad25e6c P 66114ce880744f0fb8e104a23344c94e: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 11:33:46.709138 17912 consensus_queue.cc:260] T a8299da1318f466aacf27c5a9ad25e6c P 325431e086e7403abdf19a759cd4c809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66114ce880744f0fb8e104a23344c94e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39819 } } peers { permanent_uuid: "c893fecb8c0b4bdaa6226426c4200c80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45201 } } peers { permanent_uuid: "325431e086e7403abdf19a759cd4c809" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40433 } }
I20260926 11:33:46.709185 17911 consensus_queue.cc:260] T a8299da1318f466aacf27c5a9ad25e6c P c893fecb8c0b4bdaa6226426c4200c80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66114ce880744f0fb8e104a23344c94e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39819 } } peers { permanent_uuid: "c893fecb8c0b4bdaa6226426c4200c80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45201 } } peers { permanent_uuid: "325431e086e7403abdf19a759cd4c809" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40433 } }
I20260926 11:33:46.709812 17893 heartbeater.cc:503] Master 127.0.98.62:44095 was elected leader, sending a full tablet report...
I20260926 11:33:46.710106 17911 ts_tablet_manager.cc:1428] T a8299da1318f466aacf27c5a9ad25e6c P c893fecb8c0b4bdaa6226426c4200c80: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 11:33:46.710106 17912 ts_tablet_manager.cc:1428] T a8299da1318f466aacf27c5a9ad25e6c P 325431e086e7403abdf19a759cd4c809: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 11:33:46.710440 17759 heartbeater.cc:503] Master 127.0.98.62:44095 was elected leader, sending a full tablet report...
W20260926 11:33:46.886068 17894 tablet.cc:2369] T a8299da1318f466aacf27c5a9ad25e6c P c893fecb8c0b4bdaa6226426c4200c80: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 11:33:46.903604 17760 tablet.cc:2369] T a8299da1318f466aacf27c5a9ad25e6c P 325431e086e7403abdf19a759cd4c809: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 11:33:46.929684 17626 tablet.cc:2369] T a8299da1318f466aacf27c5a9ad25e6c P 66114ce880744f0fb8e104a23344c94e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:33:46.955061 17916 raft_consensus.cc:482] T a8299da1318f466aacf27c5a9ad25e6c P 66114ce880744f0fb8e104a23344c94e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:33:46.955328 17916 raft_consensus.cc:504] T a8299da1318f466aacf27c5a9ad25e6c P 66114ce880744f0fb8e104a23344c94e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66114ce880744f0fb8e104a23344c94e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39819 } } peers { permanent_uuid: "c893fecb8c0b4bdaa6226426c4200c80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45201 } } peers { permanent_uuid: "325431e086e7403abdf19a759cd4c809" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40433 } }
I20260926 11:33:46.956508 17916 leader_election.cc:288] T a8299da1318f466aacf27c5a9ad25e6c P 66114ce880744f0fb8e104a23344c94e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c893fecb8c0b4bdaa6226426c4200c80 (127.0.98.3:45201), 325431e086e7403abdf19a759cd4c809 (127.0.98.2:40433)
I20260926 11:33:46.961644 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:46.956881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58295 (local address 127.0.98.3:45201)
0926 11:33:46.957035 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:46.957040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:46.957250 (+ 210us) server_negotiation.cc:407] Connection header received
0926 11:33:46.957460 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:46.957463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:46.957521 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:46.957602 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:46.957927 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.958704 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.960406 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.961023 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.961134 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:46.961195 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:46.961487 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 11:33:46.961547 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 11:33:46.961644 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:46.957440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44501 (local address 127.0.98.2:40433)
0926 11:33:46.957567 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:46.957573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:46.957782 (+ 209us) server_negotiation.cc:407] Connection header received
0926 11:33:46.957887 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:46.957892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:46.957966 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:46.958045 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:46.958292 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.958960 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.960416 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.961023 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.961134 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:46.961195 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:46.961428 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 11:33:46.961516 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 11:33:46.961735 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:46.957299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40433 (local address 127.0.98.1:44501)
0926 11:33:46.957693 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:46.957706 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:46.957799 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:46.958064 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:46.958066 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:46.958082 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:46.958174 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:46.958178 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.959059 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.959062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:46.960309 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:46.960319 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.961145 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.961149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:46.961274 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:46.961328 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 11:33:46.961409 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 11:33:46.961547 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":101,"thread_start_us":86,"threads_started":1}
I20260926 11:33:46.961735 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:46.956808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45201 (local address 127.0.98.1:58295)
0926 11:33:46.957110 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:46.957123 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:46.957288 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:46.957642 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:46.957646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:46.957671 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:46.957831 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:46.957841 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.958814 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.958816 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:46.960268 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:46.960279 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.961145 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.961149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:46.961354 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:46.961385 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 11:33:46.961451 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 11:33:46.961547 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":143,"thread_start_us":156,"threads_started":1}
I20260926 11:33:46.962507 17917 raft_consensus.cc:482] T a8299da1318f466aacf27c5a9ad25e6c P c893fecb8c0b4bdaa6226426c4200c80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:33:46.962486 17847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8299da1318f466aacf27c5a9ad25e6c" candidate_uuid: "66114ce880744f0fb8e104a23344c94e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c893fecb8c0b4bdaa6226426c4200c80" is_pre_election: true
I20260926 11:33:46.962705 17917 raft_consensus.cc:504] T a8299da1318f466aacf27c5a9ad25e6c P c893fecb8c0b4bdaa6226426c4200c80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66114ce880744f0fb8e104a23344c94e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39819 } } peers { permanent_uuid: "c893fecb8c0b4bdaa6226426c4200c80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45201 } } peers { permanent_uuid: "325431e086e7403abdf19a759cd4c809" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40433 } }
I20260926 11:33:46.962877 17713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8299da1318f466aacf27c5a9ad25e6c" candidate_uuid: "66114ce880744f0fb8e104a23344c94e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "325431e086e7403abdf19a759cd4c809" is_pre_election: true
I20260926 11:33:46.963120 17847 raft_consensus.cc:2454] T a8299da1318f466aacf27c5a9ad25e6c P c893fecb8c0b4bdaa6226426c4200c80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66114ce880744f0fb8e104a23344c94e in term 0.
I20260926 11:33:46.963146 17713 raft_consensus.cc:2454] T a8299da1318f466aacf27c5a9ad25e6c P 325431e086e7403abdf19a759cd4c809 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66114ce880744f0fb8e104a23344c94e in term 0.
I20260926 11:33:46.963581 17515 leader_election.cc:302] T a8299da1318f466aacf27c5a9ad25e6c P 66114ce880744f0fb8e104a23344c94e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66114ce880744f0fb8e104a23344c94e, c893fecb8c0b4bdaa6226426c4200c80; no voters:
I20260926 11:33:46.963861 17916 raft_consensus.cc:2781] T a8299da1318f466aacf27c5a9ad25e6c P 66114ce880744f0fb8e104a23344c94e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:33:46.963945 17916 raft_consensus.cc:482] T a8299da1318f466aacf27c5a9ad25e6c P 66114ce880744f0fb8e104a23344c94e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:33:46.963934 17917 leader_election.cc:288] T a8299da1318f466aacf27c5a9ad25e6c P c893fecb8c0b4bdaa6226426c4200c80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66114ce880744f0fb8e104a23344c94e (127.0.98.1:39819), 325431e086e7403abdf19a759cd4c809 (127.0.98.2:40433)
I20260926 11:33:46.963990 17916 raft_consensus.cc:3037] T a8299da1318f466aacf27c5a9ad25e6c P 66114ce880744f0fb8e104a23344c94e [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:33:46.965274 17916 raft_consensus.cc:504] T a8299da1318f466aacf27c5a9ad25e6c P 66114ce880744f0fb8e104a23344c94e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66114ce880744f0fb8e104a23344c94e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39819 } } peers { permanent_uuid: "c893fecb8c0b4bdaa6226426c4200c80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45201 } } peers { permanent_uuid: "325431e086e7403abdf19a759cd4c809" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40433 } }
I20260926 11:33:46.965585 17916 leader_election.cc:288] T a8299da1318f466aacf27c5a9ad25e6c P 66114ce880744f0fb8e104a23344c94e [CANDIDATE]: Term 1 election: Requested vote from peers c893fecb8c0b4bdaa6226426c4200c80 (127.0.98.3:45201), 325431e086e7403abdf19a759cd4c809 (127.0.98.2:40433)
I20260926 11:33:46.965852 17847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8299da1318f466aacf27c5a9ad25e6c" candidate_uuid: "66114ce880744f0fb8e104a23344c94e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c893fecb8c0b4bdaa6226426c4200c80"
I20260926 11:33:46.965938 17847 raft_consensus.cc:3037] T a8299da1318f466aacf27c5a9ad25e6c P c893fecb8c0b4bdaa6226426c4200c80 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:33:46.966388 17713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8299da1318f466aacf27c5a9ad25e6c" candidate_uuid: "66114ce880744f0fb8e104a23344c94e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "325431e086e7403abdf19a759cd4c809"
I20260926 11:33:46.966526 17713 raft_consensus.cc:3037] T a8299da1318f466aacf27c5a9ad25e6c P 325431e086e7403abdf19a759cd4c809 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:33:46.967224 17847 raft_consensus.cc:2454] T a8299da1318f466aacf27c5a9ad25e6c P c893fecb8c0b4bdaa6226426c4200c80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66114ce880744f0fb8e104a23344c94e in term 1.
I20260926 11:33:46.967547 17515 leader_election.cc:302] T a8299da1318f466aacf27c5a9ad25e6c P 66114ce880744f0fb8e104a23344c94e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66114ce880744f0fb8e104a23344c94e, c893fecb8c0b4bdaa6226426c4200c80; no voters:
I20260926 11:33:46.967821 17713 raft_consensus.cc:2454] T a8299da1318f466aacf27c5a9ad25e6c P 325431e086e7403abdf19a759cd4c809 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66114ce880744f0fb8e104a23344c94e in term 1.
I20260926 11:33:46.968299 17916 raft_consensus.cc:2781] T a8299da1318f466aacf27c5a9ad25e6c P 66114ce880744f0fb8e104a23344c94e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:33:46.968499 17916 raft_consensus.cc:686] T a8299da1318f466aacf27c5a9ad25e6c P 66114ce880744f0fb8e104a23344c94e [term 1 LEADER]: Becoming Leader. State: Replica: 66114ce880744f0fb8e104a23344c94e, State: Running, Role: LEADER
I20260926 11:33:46.968851 17916 consensus_queue.cc:237] T a8299da1318f466aacf27c5a9ad25e6c P 66114ce880744f0fb8e104a23344c94e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66114ce880744f0fb8e104a23344c94e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39819 } } peers { permanent_uuid: "c893fecb8c0b4bdaa6226426c4200c80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45201 } } peers { permanent_uuid: "325431e086e7403abdf19a759cd4c809" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40433 } }
I20260926 11:33:46.969720 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:46.964091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39819 (local address 127.0.98.3:59387)
0926 11:33:46.964210 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:46.964224 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:46.964306 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:46.966073 (+ 1767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:46.966076 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:46.966095 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:46.966217 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:46.966223 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.967241 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.967245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:46.968516 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:46.968528 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.969304 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.969308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:46.969413 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:46.969446 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 11:33:46.969525 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 11:33:46.969591 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 11:33:46.969738 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:46.965732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59387 (local address 127.0.98.1:39819)
0926 11:33:46.965826 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:46.965830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:46.965842 (+ 12us) server_negotiation.cc:407] Connection header received
0926 11:33:46.965882 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:46.965884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:46.965937 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:46.966042 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:46.966347 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.967085 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.968659 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.969187 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.969293 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:46.969352 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:46.969560 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 11:33:46.969633 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 11:33:46.970507 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:46.964366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40433 (local address 127.0.98.3:39565)
0926 11:33:46.964671 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:46.964684 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:46.964772 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:46.966241 (+ 1469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:46.966244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:46.966262 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:46.966383 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:46.966390 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.967371 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.967374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:46.968862 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:46.968870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.970098 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.970106 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:46.970210 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:46.970236 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 11:33:46.970318 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 11:33:46.970383 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":103,"threads_started":1}
I20260926 11:33:46.970502 17579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8299da1318f466aacf27c5a9ad25e6c" candidate_uuid: "c893fecb8c0b4bdaa6226426c4200c80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66114ce880744f0fb8e104a23344c94e" is_pre_election: true
I20260926 11:33:46.971362 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:46.965840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39565 (local address 127.0.98.2:40433)
0926 11:33:46.966052 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:46.966056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:46.966066 (+ 10us) server_negotiation.cc:407] Connection header received
0926 11:33:46.966101 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:46.966104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:46.966145 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:46.966216 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:46.966553 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.967228 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.969107 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.969664 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.969739 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:46.969801 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:46.971157 (+ 1356us) server_negotiation.cc:299] Negotiation successful
0926 11:33:46.971232 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260926 11:33:46.971884 17713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8299da1318f466aacf27c5a9ad25e6c" candidate_uuid: "c893fecb8c0b4bdaa6226426c4200c80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "325431e086e7403abdf19a759cd4c809" is_pre_election: true
I20260926 11:33:46.972067 17713 raft_consensus.cc:2379] T a8299da1318f466aacf27c5a9ad25e6c P 325431e086e7403abdf19a759cd4c809 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c893fecb8c0b4bdaa6226426c4200c80 in current term 1: Already voted for candidate 66114ce880744f0fb8e104a23344c94e in this term.
I20260926 11:33:46.972537 17783 leader_election.cc:302] T a8299da1318f466aacf27c5a9ad25e6c P c893fecb8c0b4bdaa6226426c4200c80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c893fecb8c0b4bdaa6226426c4200c80; no voters: 325431e086e7403abdf19a759cd4c809, 66114ce880744f0fb8e104a23344c94e
I20260926 11:33:46.972801 17917 raft_consensus.cc:2726] T a8299da1318f466aacf27c5a9ad25e6c P c893fecb8c0b4bdaa6226426c4200c80 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 11:33:46.972977 17429 catalog_manager.cc:5495] T a8299da1318f466aacf27c5a9ad25e6c P 66114ce880744f0fb8e104a23344c94e reported cstate change: term changed from 0 to 1, leader changed from <none> to 66114ce880744f0fb8e104a23344c94e (127.0.98.1). New cstate: current_term: 1 leader_uuid: "66114ce880744f0fb8e104a23344c94e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66114ce880744f0fb8e104a23344c94e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c893fecb8c0b4bdaa6226426c4200c80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "325431e086e7403abdf19a759cd4c809" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40433 } health_report { overall_health: UNKNOWN } } }
I20260926 11:33:46.985412 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:46.981788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49394 (local address 127.0.98.1:39819)
0926 11:33:46.981956 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:46.981960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:46.981974 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:33:46.982111 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:46.982119 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:46.982172 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:46.982283 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:33:46.982520 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.983038 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.983968 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:46.984194 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:46.984302 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:46.984610 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:33:46.985026 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:33:46.985159 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:46.985219 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 11:33:46.985272 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 11:33:46.991386 17847 raft_consensus.cc:1261] T a8299da1318f466aacf27c5a9ad25e6c P c893fecb8c0b4bdaa6226426c4200c80 [term 1 FOLLOWER]: Refusing update from remote peer 66114ce880744f0fb8e104a23344c94e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:33:46.991398 17713 raft_consensus.cc:1261] T a8299da1318f466aacf27c5a9ad25e6c P 325431e086e7403abdf19a759cd4c809 [term 1 FOLLOWER]: Refusing update from remote peer 66114ce880744f0fb8e104a23344c94e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:33:46.992113 17922 consensus_queue.cc:1035] T a8299da1318f466aacf27c5a9ad25e6c P 66114ce880744f0fb8e104a23344c94e [LEADER]: Connected to new peer: Peer: permanent_uuid: "325431e086e7403abdf19a759cd4c809" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:33:46.992328 17916 consensus_queue.cc:1035] T a8299da1318f466aacf27c5a9ad25e6c P 66114ce880744f0fb8e104a23344c94e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c893fecb8c0b4bdaa6226426c4200c80" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:33:46.999948 17927 mvcc.cc:205] Tried to move back new op lower bound from 7333570260948717568 to 7333570260867428352. Current Snapshot: MvccSnapshot[applied={T|T < 7333570260948717568}]
I20260926 11:33:47.002015 17926 mvcc.cc:205] Tried to move back new op lower bound from 7333570260948717568 to 7333570260867428352. Current Snapshot: MvccSnapshot[applied={T|T < 7333570260948717568}]
I20260926 11:33:47.007273 17430 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:33:47.007452 17430 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34690:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:33:47.009822 17430 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9a22bef8fb8045d2991b585c9fbfa4fd: Sending DeleteTablet for 3 replicas of tablet a8299da1318f466aacf27c5a9ad25e6c
I20260926 11:33:47.010581 17559 tablet_service.cc:1514] Processing DeleteTablet for tablet a8299da1318f466aacf27c5a9ad25e6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:33:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49388
I20260926 11:33:47.010627 17827 tablet_service.cc:1514] Processing DeleteTablet for tablet a8299da1318f466aacf27c5a9ad25e6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:33:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50696
I20260926 11:33:47.011059 17936 tablet_replica.cc:330] T a8299da1318f466aacf27c5a9ad25e6c P 66114ce880744f0fb8e104a23344c94e: stopping tablet replica
I20260926 11:33:47.011078 17692 tablet_service.cc:1514] Processing DeleteTablet for tablet a8299da1318f466aacf27c5a9ad25e6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:33:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49700
I20260926 11:33:47.011355 17936 raft_consensus.cc:2229] T a8299da1318f466aacf27c5a9ad25e6c P 66114ce880744f0fb8e104a23344c94e [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:33:47.011605 17937 tablet_replica.cc:330] T a8299da1318f466aacf27c5a9ad25e6c P 325431e086e7403abdf19a759cd4c809: stopping tablet replica
I20260926 11:33:47.011765 17936 raft_consensus.cc:2258] T a8299da1318f466aacf27c5a9ad25e6c P 66114ce880744f0fb8e104a23344c94e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:33:47.012118 17937 raft_consensus.cc:2229] T a8299da1318f466aacf27c5a9ad25e6c P 325431e086e7403abdf19a759cd4c809 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:33:47.012342 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 17496
I20260926 11:33:47.013943 17937 raft_consensus.cc:2258] T a8299da1318f466aacf27c5a9ad25e6c P 325431e086e7403abdf19a759cd4c809 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:33:47.014307 17935 tablet_replica.cc:330] T a8299da1318f466aacf27c5a9ad25e6c P c893fecb8c0b4bdaa6226426c4200c80: stopping tablet replica
I20260926 11:33:47.014614 17935 raft_consensus.cc:2229] T a8299da1318f466aacf27c5a9ad25e6c P c893fecb8c0b4bdaa6226426c4200c80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:33:47.014915 17935 raft_consensus.cc:2258] T a8299da1318f466aacf27c5a9ad25e6c P c893fecb8c0b4bdaa6226426c4200c80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:33:47.015420 17937 ts_tablet_manager.cc:1905] T a8299da1318f466aacf27c5a9ad25e6c P 325431e086e7403abdf19a759cd4c809: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:33:47.016439 17935 ts_tablet_manager.cc:1905] T a8299da1318f466aacf27c5a9ad25e6c P c893fecb8c0b4bdaa6226426c4200c80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:33:47.017724 17937 ts_tablet_manager.cc:1918] T a8299da1318f466aacf27c5a9ad25e6c P 325431e086e7403abdf19a759cd4c809: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:33:47.017832 17937 log.cc:1196] T a8299da1318f466aacf27c5a9ad25e6c P 325431e086e7403abdf19a759cd4c809: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-1/wal/wals/a8299da1318f466aacf27c5a9ad25e6c
I20260926 11:33:47.018169 17937 ts_tablet_manager.cc:1939] T a8299da1318f466aacf27c5a9ad25e6c P 325431e086e7403abdf19a759cd4c809: Deleting consensus metadata
I20260926 11:33:47.019402 17421 catalog_manager.cc:4841] TS 325431e086e7403abdf19a759cd4c809 (127.0.98.2:40433): tablet a8299da1318f466aacf27c5a9ad25e6c (table test-table [id=7185b78dcfca4988b26262ce7ebed527]) successfully deleted
I20260926 11:33:47.019829 17935 ts_tablet_manager.cc:1918] T a8299da1318f466aacf27c5a9ad25e6c P c893fecb8c0b4bdaa6226426c4200c80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:33:47.019930 17935 log.cc:1196] T a8299da1318f466aacf27c5a9ad25e6c P c893fecb8c0b4bdaa6226426c4200c80: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790422180210734-392-0/minicluster-data/ts-2/wal/wals/a8299da1318f466aacf27c5a9ad25e6c
I20260926 11:33:47.020287 17935 ts_tablet_manager.cc:1939] T a8299da1318f466aacf27c5a9ad25e6c P c893fecb8c0b4bdaa6226426c4200c80: Deleting consensus metadata
W20260926 11:33:47.020292 17420 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39819 (error 108)
I20260926 11:33:47.020520 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 17630
I20260926 11:33:47.022048 17421 catalog_manager.cc:4841] TS c893fecb8c0b4bdaa6226426c4200c80 (127.0.98.3:45201): tablet a8299da1318f466aacf27c5a9ad25e6c (table test-table [id=7185b78dcfca4988b26262ce7ebed527]) successfully deleted
I20260926 11:33:47.022466 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:47.022154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39819 (local address 127.0.0.1:49402)
0926 11:33:47.022313 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:47.022387 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39819: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260926 11:33:47.022629 17420 catalog_manager.cc:4584] TS 66114ce880744f0fb8e104a23344c94e (127.0.98.1:39819): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a8299da1318f466aacf27c5a9ad25e6c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39819: connect: Connection refused (error 111)
I20260926 11:33:47.026651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 17764
I20260926 11:33:47.032335 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 17402
2026-09-26T11:33:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2489 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T11:33:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:33:47Z Disabled control of system clock
I20260926 11:33:47.063542 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40687
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 11:33:47.174031 17942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:47.174317 17942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:47.174362 17942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:47.178568 17942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:33:47.178634 17942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:47.178650 17942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:47.178668 17942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:33:47.178683 17942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:33:47.183561 17942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40687
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:47.184731 17942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:47.186084 17942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:33:47.195089 17948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:47.195190 17950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:47.195113 17947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:47.195537 17942 server_base.cc:1034] running on GCE node
I20260926 11:33:47.196214 17942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:47.197371 17942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:47.198544 17942 hybrid_clock.cc:648] HybridClock initialized: now 1790422427198515 us; error 46 us; skew 500 ppm
I20260926 11:33:47.200609 17942 webserver.cc:460] Webserver started at http://127.0.98.62:39375/ using document root <none> and password file <none>
I20260926 11:33:47.201176 17942 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:47.201288 17942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:47.201490 17942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:33:47.203696 17942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "3b7d7f52a9db4aa2952797ad27c60d88"
format_stamp: "Formatted at 2026-09-26 11:33:47 on dist-test-slave-mrck"
server_key: "05434db4fe5a32b11e8665882a796b9f"
server_key_iv: "bb26b1fec3b5b49a6ad1de444a4b8325"
server_key_version: "encryptionkey@0"
I20260926 11:33:47.204319 17942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "3b7d7f52a9db4aa2952797ad27c60d88"
format_stamp: "Formatted at 2026-09-26 11:33:47 on dist-test-slave-mrck"
server_key: "05434db4fe5a32b11e8665882a796b9f"
server_key_iv: "bb26b1fec3b5b49a6ad1de444a4b8325"
server_key_version: "encryptionkey@0"
I20260926 11:33:47.208173 17942 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 11:33:47.210723 17956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:47.212065 17942 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:33:47.212168 17942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "3b7d7f52a9db4aa2952797ad27c60d88"
format_stamp: "Formatted at 2026-09-26 11:33:47 on dist-test-slave-mrck"
server_key: "05434db4fe5a32b11e8665882a796b9f"
server_key_iv: "bb26b1fec3b5b49a6ad1de444a4b8325"
server_key_version: "encryptionkey@0"
I20260926 11:33:47.212256 17942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:47.236771 17942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:47.237489 17942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:47.237658 17942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:47.245105 18008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40687 every 8 connection(s)
I20260926 11:33:47.245124 17942 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40687
I20260926 11:33:47.246170 17942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:33:47.249295 18009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:33:47.249708 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 17942
I20260926 11:33:47.249842 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:33:47.250159 392 external_mini_cluster.cc:1428] Setting key 2f69679ed470189b34ac4fa2005341b5
I20260926 11:33:47.256268 18009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b7d7f52a9db4aa2952797ad27c60d88: Bootstrap starting.
I20260926 11:33:47.258882 18009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b7d7f52a9db4aa2952797ad27c60d88: Neither blocks nor log segments found. Creating new log.
I20260926 11:33:47.259850 18009 log.cc:824] T 00000000000000000000000000000000 P 3b7d7f52a9db4aa2952797ad27c60d88: Log is configured to *not* fsync() on all Append() calls
I20260926 11:33:47.260835 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:47.251583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47968 (local address 127.0.98.62:40687)
0926 11:33:47.252326 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:47.252343 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:47.252389 (+ 46us) server_negotiation.cc:407] Connection header received
0926 11:33:47.253262 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:47.253285 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:47.253797 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:47.254221 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:47.254686 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.256197 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.257096 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.257790 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.257955 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:47.258727 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:47.258735 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:47.258753 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:33:47.259288 (+ 535us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:47.259357 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:33:47.259383 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:47.259487 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:47.259630 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 11:33:47.259938 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":157,"threads_started":1}
I20260926 11:33:47.261966 18009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b7d7f52a9db4aa2952797ad27c60d88: No bootstrap required, opened a new log
I20260926 11:33:47.265614 18009 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b7d7f52a9db4aa2952797ad27c60d88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7d7f52a9db4aa2952797ad27c60d88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40687 } }
I20260926 11:33:47.265826 18009 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b7d7f52a9db4aa2952797ad27c60d88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:33:47.265906 18009 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b7d7f52a9db4aa2952797ad27c60d88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b7d7f52a9db4aa2952797ad27c60d88, State: Initialized, Role: FOLLOWER
I20260926 11:33:47.266302 18009 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b7d7f52a9db4aa2952797ad27c60d88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7d7f52a9db4aa2952797ad27c60d88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40687 } }
I20260926 11:33:47.266458 18009 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b7d7f52a9db4aa2952797ad27c60d88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:33:47.266547 18009 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b7d7f52a9db4aa2952797ad27c60d88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:33:47.266685 18009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b7d7f52a9db4aa2952797ad27c60d88 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:33:47.267657 18009 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b7d7f52a9db4aa2952797ad27c60d88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7d7f52a9db4aa2952797ad27c60d88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40687 } }
I20260926 11:33:47.268009 18009 leader_election.cc:302] T 00000000000000000000000000000000 P 3b7d7f52a9db4aa2952797ad27c60d88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b7d7f52a9db4aa2952797ad27c60d88; no voters:
I20260926 11:33:47.268304 18009 leader_election.cc:288] T 00000000000000000000000000000000 P 3b7d7f52a9db4aa2952797ad27c60d88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:33:47.268446 18014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b7d7f52a9db4aa2952797ad27c60d88 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:33:47.268667 18014 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b7d7f52a9db4aa2952797ad27c60d88 [term 1 LEADER]: Becoming Leader. State: Replica: 3b7d7f52a9db4aa2952797ad27c60d88, State: Running, Role: LEADER
I20260926 11:33:47.268949 18014 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b7d7f52a9db4aa2952797ad27c60d88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7d7f52a9db4aa2952797ad27c60d88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40687 } }
I20260926 11:33:47.269492 18009 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b7d7f52a9db4aa2952797ad27c60d88 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:33:47.270542 18016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b7d7f52a9db4aa2952797ad27c60d88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b7d7f52a9db4aa2952797ad27c60d88. Latest consensus state: current_term: 1 leader_uuid: "3b7d7f52a9db4aa2952797ad27c60d88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7d7f52a9db4aa2952797ad27c60d88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40687 } } }
I20260926 11:33:47.270671 18016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b7d7f52a9db4aa2952797ad27c60d88 [sys.catalog]: This master's current role is: LEADER
I20260926 11:33:47.270964 18015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b7d7f52a9db4aa2952797ad27c60d88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b7d7f52a9db4aa2952797ad27c60d88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b7d7f52a9db4aa2952797ad27c60d88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40687 } } }
I20260926 11:33:47.271057 18015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b7d7f52a9db4aa2952797ad27c60d88 [sys.catalog]: This master's current role is: LEADER
I20260926 11:33:47.271165 18023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:33:47.274353 18023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:33:47.280103 18023 catalog_manager.cc:1347] Generated new cluster ID: b89b6bdf4415429bb8a1dcfe7e5becac
I20260926 11:33:47.280181 18023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:33:47.302044 18023 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:33:47.303355 18023 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:33:47.317144 18023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b7d7f52a9db4aa2952797ad27c60d88: Generated new TSK 0
I20260926 11:33:47.317979 18023 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:33:47.328778 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40687
--builtin_ntp_servers=127.0.98.20:38389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 11:33:47.442211 18033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:47.442516 18033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:47.442586 18033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:47.446672 18033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:47.446777 18033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:47.446879 18033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:33:47.451567 18033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:47.452679 18033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:47.454094 18033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:33:47.463565 18038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:47.463826 18033 server_base.cc:1034] running on GCE node
W20260926 11:33:47.463565 18041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:47.463572 18039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:47.464408 18033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:47.464984 18033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:47.466173 18033 hybrid_clock.cc:648] HybridClock initialized: now 1790422427466159 us; error 30 us; skew 500 ppm
I20260926 11:33:47.467984 18033 webserver.cc:460] Webserver started at http://127.0.98.1:34403/ using document root <none> and password file <none>
I20260926 11:33:47.468551 18033 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:47.468644 18033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:47.468868 18033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:33:47.471081 18033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-0/data/instance:
uuid: "57995f568fe8413896c2592d3b329be2"
format_stamp: "Formatted at 2026-09-26 11:33:47 on dist-test-slave-mrck"
server_key: "c95bd310a9a4d650f35df85db2414806"
server_key_iv: "102bb4ad5fd4159de7212539d307a1c6"
server_key_version: "encryptionkey@0"
I20260926 11:33:47.471661 18033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "57995f568fe8413896c2592d3b329be2"
format_stamp: "Formatted at 2026-09-26 11:33:47 on dist-test-slave-mrck"
server_key: "c95bd310a9a4d650f35df85db2414806"
server_key_iv: "102bb4ad5fd4159de7212539d307a1c6"
server_key_version: "encryptionkey@0"
I20260926 11:33:47.475252 18033 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:33:47.477648 18047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:47.478833 18033 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:33:47.478969 18033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "57995f568fe8413896c2592d3b329be2"
format_stamp: "Formatted at 2026-09-26 11:33:47 on dist-test-slave-mrck"
server_key: "c95bd310a9a4d650f35df85db2414806"
server_key_iv: "102bb4ad5fd4159de7212539d307a1c6"
server_key_version: "encryptionkey@0"
I20260926 11:33:47.479086 18033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:47.493018 18033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:47.493696 18033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:47.493888 18033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:47.494469 18033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:33:47.495370 18033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:33:47.495442 18033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:47.495523 18033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:33:47.495580 18033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:47.504452 18033 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44671
I20260926 11:33:47.504477 18160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44671 every 8 connection(s)
I20260926 11:33:47.505345 18033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:33:47.514403 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:47.507125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39233 (local address 127.0.98.62:40687)
0926 11:33:47.507271 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:47.507275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:47.508325 (+ 1050us) server_negotiation.cc:407] Connection header received
0926 11:33:47.509709 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:47.509712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:47.509770 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:47.509853 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:47.510725 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.511278 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.512468 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.512689 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.512795 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:47.513878 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:47.513882 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:47.513885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:33:47.513897 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:47.513925 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:33:47.513932 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:47.514026 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:47.514183 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 11:33:47.514245 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 11:33:47.515223 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 18033
I20260926 11:33:47.515359 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:33:47.515388 18163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:47.507529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40687 (local address 127.0.98.1:39233)
0926 11:33:47.508170 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:47.508209 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:47.509494 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:47.510020 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:47.510027 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:47.510112 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:33:47.510585 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:47.510598 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.511410 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.511418 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:47.512347 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:47.512356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.512834 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.512838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:47.512997 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:47.513606 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:33:47.513608 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:33:47.513646 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:33:47.513715 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:33:47.514019 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:33:47.514028 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:33:47.514036 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 11:33:47.514174 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 11:33:47.514481 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":147,"threads_started":1}
I20260926 11:33:47.515781 392 external_mini_cluster.cc:1428] Setting key e371f93a838efc7ad977d277986b622c
I20260926 11:33:47.516700 18161 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40687
I20260926 11:33:47.516949 18161 heartbeater.cc:464] Registering TS with master...
I20260926 11:33:47.517493 18161 heartbeater.cc:511] Master 127.0.98.62:40687 requested a full tablet report, sending...
I20260926 11:33:47.517675 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40687
--builtin_ntp_servers=127.0.98.20:38389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 11:33:47.519047 17973 ts_manager.cc:194] Registered new tserver with Master: 57995f568fe8413896c2592d3b329be2 (127.0.98.1:44671)
I20260926 11:33:47.520279 17973 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:39233
W20260926 11:33:47.628573 18164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:47.628859 18164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:47.628899 18164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:47.633095 18164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:47.633164 18164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:47.633221 18164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:33:47.638149 18164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:47.639281 18164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:47.640698 18164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:33:47.650404 18170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:47.650496 18172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:47.650511 18169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:47.650646 18164 server_base.cc:1034] running on GCE node
I20260926 11:33:47.651117 18164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:47.651791 18164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:47.653038 18164 hybrid_clock.cc:648] HybridClock initialized: now 1790422427652999 us; error 51 us; skew 500 ppm
I20260926 11:33:47.655102 18164 webserver.cc:460] Webserver started at http://127.0.98.2:44091/ using document root <none> and password file <none>
I20260926 11:33:47.655761 18164 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:47.655865 18164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:47.656095 18164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:33:47.658171 18164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-1/data/instance:
uuid: "c878fd758eff4fc5a0798609b8c74bd1"
format_stamp: "Formatted at 2026-09-26 11:33:47 on dist-test-slave-mrck"
server_key: "6ae94003e175f30619c337a8f7987641"
server_key_iv: "684ffa9be32189d90bac2b062f588ae7"
server_key_version: "encryptionkey@0"
I20260926 11:33:47.658741 18164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c878fd758eff4fc5a0798609b8c74bd1"
format_stamp: "Formatted at 2026-09-26 11:33:47 on dist-test-slave-mrck"
server_key: "6ae94003e175f30619c337a8f7987641"
server_key_iv: "684ffa9be32189d90bac2b062f588ae7"
server_key_version: "encryptionkey@0"
I20260926 11:33:47.662783 18164 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 11:33:47.665336 18178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:47.666535 18164 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 11:33:47.666682 18164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "c878fd758eff4fc5a0798609b8c74bd1"
format_stamp: "Formatted at 2026-09-26 11:33:47 on dist-test-slave-mrck"
server_key: "6ae94003e175f30619c337a8f7987641"
server_key_iv: "684ffa9be32189d90bac2b062f588ae7"
server_key_version: "encryptionkey@0"
I20260926 11:33:47.666800 18164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:47.679216 18164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:47.680048 18164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:47.680256 18164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:47.680867 18164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:33:47.681885 18164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:33:47.681953 18164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:47.682022 18164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:33:47.682070 18164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:47.690858 18164 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43755
I20260926 11:33:47.690925 18291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43755 every 8 connection(s)
I20260926 11:33:47.691869 18164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:33:47.694700 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 18164
I20260926 11:33:47.694870 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:33:47.695178 392 external_mini_cluster.cc:1428] Setting key 40c36a29cb5fd92c33e91d82ddb25c6b
I20260926 11:33:47.697458 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40687
--builtin_ntp_servers=127.0.98.20:38389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 11:33:47.703126 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:47.693844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44053 (local address 127.0.98.62:40687)
0926 11:33:47.694016 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:47.694020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:47.695203 (+ 1183us) server_negotiation.cc:407] Connection header received
0926 11:33:47.697005 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:47.697009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:47.697061 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:47.697151 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:47.698130 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.698955 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.700841 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.701066 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.701155 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:47.702551 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:47.702554 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:47.702556 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:33:47.702568 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:47.702607 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:33:47.702618 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:47.702696 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:47.702891 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 11:33:47.702982 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 11:33:47.704154 18294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:47.694289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40687 (local address 127.0.98.2:44053)
0926 11:33:47.695017 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:47.695070 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:47.696766 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:47.697309 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:47.697316 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:47.697402 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:33:47.697989 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:47.698002 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.699092 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.699098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:47.700712 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:47.700726 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.701239 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.701245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:47.701502 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:47.702187 (+ 685us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:33:47.702191 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:33:47.702268 (+ 77us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:33:47.702366 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:33:47.702730 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:33:47.702739 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:33:47.702745 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 11:33:47.702892 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 11:33:47.703181 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":139,"threads_started":1}
I20260926 11:33:47.705327 18292 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40687
I20260926 11:33:47.705612 18292 heartbeater.cc:464] Registering TS with master...
I20260926 11:33:47.706284 18292 heartbeater.cc:511] Master 127.0.98.62:40687 requested a full tablet report, sending...
I20260926 11:33:47.707458 17973 ts_manager.cc:194] Registered new tserver with Master: c878fd758eff4fc5a0798609b8c74bd1 (127.0.98.2:43755)
I20260926 11:33:47.708073 17973 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:44053
W20260926 11:33:47.810001 18295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:47.810280 18295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:47.810318 18295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:47.814420 18295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:47.814478 18295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:47.814577 18295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:33:47.819607 18295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:47.820699 18295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:47.822060 18295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:33:47.831254 18300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:47.831619 18301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:47.831676 18303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:47.831756 18295 server_base.cc:1034] running on GCE node
I20260926 11:33:47.832157 18295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:47.832831 18295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:47.833999 18295 hybrid_clock.cc:648] HybridClock initialized: now 1790422427833973 us; error 46 us; skew 500 ppm
I20260926 11:33:47.835976 18295 webserver.cc:460] Webserver started at http://127.0.98.3:44325/ using document root <none> and password file <none>
I20260926 11:33:47.836581 18295 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:47.836647 18295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:47.836905 18295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:33:47.839052 18295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-2/data/instance:
uuid: "8436329f0c7c4ab9a0b2afeb7ff3c8c5"
format_stamp: "Formatted at 2026-09-26 11:33:47 on dist-test-slave-mrck"
server_key: "089aae26aab143b21e3ad519c616eb7f"
server_key_iv: "32f300f03c4827b6b3fed3ed049a37b0"
server_key_version: "encryptionkey@0"
I20260926 11:33:47.839635 18295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8436329f0c7c4ab9a0b2afeb7ff3c8c5"
format_stamp: "Formatted at 2026-09-26 11:33:47 on dist-test-slave-mrck"
server_key: "089aae26aab143b21e3ad519c616eb7f"
server_key_iv: "32f300f03c4827b6b3fed3ed049a37b0"
server_key_version: "encryptionkey@0"
I20260926 11:33:47.843252 18295 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:33:47.845680 18309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:47.846989 18295 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 11:33:47.847123 18295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "8436329f0c7c4ab9a0b2afeb7ff3c8c5"
format_stamp: "Formatted at 2026-09-26 11:33:47 on dist-test-slave-mrck"
server_key: "089aae26aab143b21e3ad519c616eb7f"
server_key_iv: "32f300f03c4827b6b3fed3ed049a37b0"
server_key_version: "encryptionkey@0"
I20260926 11:33:47.847225 18295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:47.863474 18295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:47.864245 18295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:47.864444 18295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:47.865042 18295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:33:47.865976 18295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:33:47.866031 18295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:47.866102 18295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:33:47.866146 18295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:47.875690 18295 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46065
I20260926 11:33:47.875712 18422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46065 every 8 connection(s)
I20260926 11:33:47.876708 18295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:33:47.884143 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 18295
I20260926 11:33:47.884261 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:33:47.884545 392 external_mini_cluster.cc:1428] Setting key 22b0840c809b69983410ff33ec3cc155
I20260926 11:33:47.886404 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:47.878590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47185 (local address 127.0.98.62:40687)
0926 11:33:47.878734 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:47.878737 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:47.879820 (+ 1083us) server_negotiation.cc:407] Connection header received
0926 11:33:47.881181 (+ 1361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:47.881183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:47.881230 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:47.881305 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:47.882341 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.882954 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.884407 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.884653 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.884759 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:47.885909 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:47.885912 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:47.885915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:33:47.885929 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:47.885957 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:33:47.885965 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:47.886043 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:47.886196 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 11:33:47.886256 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 11:33:47.887269 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:47.878993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40687 (local address 127.0.98.3:47185)
0926 11:33:47.879647 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:47.879687 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:47.880990 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:47.881464 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:47.881471 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:47.881555 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:33:47.882129 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:47.882152 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.883091 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.883094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:47.884209 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:47.884220 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.884838 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.884842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:47.885026 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:47.885655 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:33:47.885657 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:33:47.885697 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:33:47.885760 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:33:47.886049 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:33:47.886062 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:33:47.886067 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 11:33:47.886171 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 11:33:47.886401 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":117,"threads_started":1}
I20260926 11:33:47.888525 18423 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40687
I20260926 11:33:47.888803 18423 heartbeater.cc:464] Registering TS with master...
I20260926 11:33:47.889367 18423 heartbeater.cc:511] Master 127.0.98.62:40687 requested a full tablet report, sending...
I20260926 11:33:47.890475 17973 ts_manager.cc:194] Registered new tserver with Master: 8436329f0c7c4ab9a0b2afeb7ff3c8c5 (127.0.98.3:46065)
I20260926 11:33:47.891068 17973 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:47185
I20260926 11:33:47.899154 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:33:47.906359 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:47.902479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47984 (local address 127.0.98.62:40687)
0926 11:33:47.902651 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:47.902657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:47.902987 (+ 330us) server_negotiation.cc:407] Connection header received
0926 11:33:47.903135 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:47.903138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:47.903194 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:47.903264 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:47.903560 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.904145 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.904974 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.905183 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.905258 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:47.905950 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:47.905953 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:47.905955 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:33:47.905965 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:47.905982 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:33:47.905990 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:47.906046 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:47.906123 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 11:33:47.906173 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 11:33:47.909085 17973 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:33:47.912074 17973 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:33:47.926035 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:47.920396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34038 (local address 127.0.98.1:44671)
0926 11:33:47.920824 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:47.920838 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:47.920866 (+ 28us) server_negotiation.cc:407] Connection header received
0926 11:33:47.920960 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:47.920965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:47.921209 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:47.921395 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:47.921900 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.923029 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.924676 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.925380 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.925495 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:47.925686 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:47.925782 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 11:33:47.925875 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":137,"threads_started":1}
I20260926 11:33:47.926024 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:47.920208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44671 (local address 127.0.0.1:34038)
0926 11:33:47.920562 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:47.920614 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:47.920744 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:47.921574 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:47.921580 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:47.921601 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:47.921737 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:47.921745 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.923144 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.923147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:47.924568 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:47.924578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.925505 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.925508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:47.925606 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:47.925641 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 11:33:47.925752 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 11:33:47.925868 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":97,"threads_started":1}
I20260926 11:33:47.926609 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:47.920600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46065 (local address 127.0.0.1:53826)
0926 11:33:47.921745 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:47.921757 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:47.921850 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:47.922319 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:47.922322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:47.922337 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:47.922429 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:47.922435 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.923655 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.923659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:47.924866 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:47.924872 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.926134 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.926138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:47.926254 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:47.926281 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 11:33:47.926362 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 11:33:47.926444 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1075,"mutex_wait_us":92,"thread_start_us":110,"threads_started":1}
I20260926 11:33:47.926621 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:47.921069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53826 (local address 127.0.98.3:46065)
0926 11:33:47.921563 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:47.921572 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:47.921831 (+ 259us) server_negotiation.cc:407] Connection header received
0926 11:33:47.921980 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:47.921986 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:47.922187 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:47.922346 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:47.922556 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.923518 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.924996 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.925793 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.925898 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:47.926067 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:47.926386 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 11:33:47.926478 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":133,"threads_started":1}
I20260926 11:33:47.928026 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:47.921124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55588 (local address 127.0.98.2:43755)
0926 11:33:47.921524 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:47.921536 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:47.922133 (+ 597us) server_negotiation.cc:407] Connection header received
0926 11:33:47.922215 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:47.922220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:47.922394 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:47.922533 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:47.922722 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.923585 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.925088 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.925797 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.925898 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:47.926067 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:47.927826 (+ 1759us) server_negotiation.cc:299] Negotiation successful
0926 11:33:47.927904 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":102,"threads_started":1}
I20260926 11:33:47.928222 18436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:47.920965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43755 (local address 127.0.0.1:55588)
0926 11:33:47.922013 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:47.922025 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:47.922111 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:47.922514 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:47.922519 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:47.922539 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:47.922612 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:47.922617 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.923695 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.923702 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:47.924947 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:47.924956 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.927383 (+ 2427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.927387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:47.927472 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:47.927494 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 11:33:47.927567 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 11:33:47.927644 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":982,"mutex_wait_us":23,"thread_start_us":93,"threads_started":1}
I20260926 11:33:47.929279 18095 tablet_service.cc:1467] Processing CreateTablet for tablet 7c1c551449ed4512a7a76e4247193921 (DEFAULT_TABLE table=test-table [id=d34fe65777e54887999f66d7fe9fbd5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:33:47.929279 18357 tablet_service.cc:1467] Processing CreateTablet for tablet 7c1c551449ed4512a7a76e4247193921 (DEFAULT_TABLE table=test-table [id=d34fe65777e54887999f66d7fe9fbd5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:33:47.930404 18095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c1c551449ed4512a7a76e4247193921. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:33:47.930404 18357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c1c551449ed4512a7a76e4247193921. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:33:47.930536 18226 tablet_service.cc:1467] Processing CreateTablet for tablet 7c1c551449ed4512a7a76e4247193921 (DEFAULT_TABLE table=test-table [id=d34fe65777e54887999f66d7fe9fbd5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:33:47.931449 18226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c1c551449ed4512a7a76e4247193921. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:33:47.937373 18440 tablet_bootstrap.cc:492] T 7c1c551449ed4512a7a76e4247193921 P c878fd758eff4fc5a0798609b8c74bd1: Bootstrap starting.
I20260926 11:33:47.938455 18442 tablet_bootstrap.cc:492] T 7c1c551449ed4512a7a76e4247193921 P 57995f568fe8413896c2592d3b329be2: Bootstrap starting.
I20260926 11:33:47.939935 18440 tablet_bootstrap.cc:654] T 7c1c551449ed4512a7a76e4247193921 P c878fd758eff4fc5a0798609b8c74bd1: Neither blocks nor log segments found. Creating new log.
I20260926 11:33:47.940057 18441 tablet_bootstrap.cc:492] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5: Bootstrap starting.
I20260926 11:33:47.940568 18442 tablet_bootstrap.cc:654] T 7c1c551449ed4512a7a76e4247193921 P 57995f568fe8413896c2592d3b329be2: Neither blocks nor log segments found. Creating new log.
I20260926 11:33:47.940841 18440 log.cc:824] T 7c1c551449ed4512a7a76e4247193921 P c878fd758eff4fc5a0798609b8c74bd1: Log is configured to *not* fsync() on all Append() calls
I20260926 11:33:47.941347 18442 log.cc:824] T 7c1c551449ed4512a7a76e4247193921 P 57995f568fe8413896c2592d3b329be2: Log is configured to *not* fsync() on all Append() calls
I20260926 11:33:47.941942 18441 tablet_bootstrap.cc:654] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5: Neither blocks nor log segments found. Creating new log.
I20260926 11:33:47.942746 18441 log.cc:824] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5: Log is configured to *not* fsync() on all Append() calls
I20260926 11:33:47.943073 18440 tablet_bootstrap.cc:492] T 7c1c551449ed4512a7a76e4247193921 P c878fd758eff4fc5a0798609b8c74bd1: No bootstrap required, opened a new log
I20260926 11:33:47.943264 18440 ts_tablet_manager.cc:1397] T 7c1c551449ed4512a7a76e4247193921 P c878fd758eff4fc5a0798609b8c74bd1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 11:33:47.943343 18442 tablet_bootstrap.cc:492] T 7c1c551449ed4512a7a76e4247193921 P 57995f568fe8413896c2592d3b329be2: No bootstrap required, opened a new log
I20260926 11:33:47.943528 18442 ts_tablet_manager.cc:1397] T 7c1c551449ed4512a7a76e4247193921 P 57995f568fe8413896c2592d3b329be2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 11:33:47.944486 18441 tablet_bootstrap.cc:492] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5: No bootstrap required, opened a new log
I20260926 11:33:47.944672 18441 ts_tablet_manager.cc:1397] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 11:33:47.947588 18441 raft_consensus.cc:348] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57995f568fe8413896c2592d3b329be2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44671 } } peers { permanent_uuid: "c878fd758eff4fc5a0798609b8c74bd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43755 } } peers { permanent_uuid: "8436329f0c7c4ab9a0b2afeb7ff3c8c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46065 } }
I20260926 11:33:47.947635 18442 raft_consensus.cc:348] T 7c1c551449ed4512a7a76e4247193921 P 57995f568fe8413896c2592d3b329be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57995f568fe8413896c2592d3b329be2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44671 } } peers { permanent_uuid: "c878fd758eff4fc5a0798609b8c74bd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43755 } } peers { permanent_uuid: "8436329f0c7c4ab9a0b2afeb7ff3c8c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46065 } }
I20260926 11:33:47.947635 18440 raft_consensus.cc:348] T 7c1c551449ed4512a7a76e4247193921 P c878fd758eff4fc5a0798609b8c74bd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57995f568fe8413896c2592d3b329be2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44671 } } peers { permanent_uuid: "c878fd758eff4fc5a0798609b8c74bd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43755 } } peers { permanent_uuid: "8436329f0c7c4ab9a0b2afeb7ff3c8c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46065 } }
I20260926 11:33:47.947851 18441 raft_consensus.cc:374] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:33:47.947901 18442 raft_consensus.cc:374] T 7c1c551449ed4512a7a76e4247193921 P 57995f568fe8413896c2592d3b329be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:33:47.947901 18440 raft_consensus.cc:374] T 7c1c551449ed4512a7a76e4247193921 P c878fd758eff4fc5a0798609b8c74bd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:33:47.947909 18441 raft_consensus.cc:729] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8436329f0c7c4ab9a0b2afeb7ff3c8c5, State: Initialized, Role: FOLLOWER
I20260926 11:33:47.947953 18442 raft_consensus.cc:729] T 7c1c551449ed4512a7a76e4247193921 P 57995f568fe8413896c2592d3b329be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57995f568fe8413896c2592d3b329be2, State: Initialized, Role: FOLLOWER
I20260926 11:33:47.947959 18440 raft_consensus.cc:729] T 7c1c551449ed4512a7a76e4247193921 P c878fd758eff4fc5a0798609b8c74bd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c878fd758eff4fc5a0798609b8c74bd1, State: Initialized, Role: FOLLOWER
I20260926 11:33:47.948365 18441 consensus_queue.cc:260] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57995f568fe8413896c2592d3b329be2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44671 } } peers { permanent_uuid: "c878fd758eff4fc5a0798609b8c74bd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43755 } } peers { permanent_uuid: "8436329f0c7c4ab9a0b2afeb7ff3c8c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46065 } }
I20260926 11:33:47.948403 18440 consensus_queue.cc:260] T 7c1c551449ed4512a7a76e4247193921 P c878fd758eff4fc5a0798609b8c74bd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57995f568fe8413896c2592d3b329be2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44671 } } peers { permanent_uuid: "c878fd758eff4fc5a0798609b8c74bd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43755 } } peers { permanent_uuid: "8436329f0c7c4ab9a0b2afeb7ff3c8c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46065 } }
I20260926 11:33:47.948403 18442 consensus_queue.cc:260] T 7c1c551449ed4512a7a76e4247193921 P 57995f568fe8413896c2592d3b329be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57995f568fe8413896c2592d3b329be2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44671 } } peers { permanent_uuid: "c878fd758eff4fc5a0798609b8c74bd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43755 } } peers { permanent_uuid: "8436329f0c7c4ab9a0b2afeb7ff3c8c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46065 } }
I20260926 11:33:47.949326 18440 ts_tablet_manager.cc:1428] T 7c1c551449ed4512a7a76e4247193921 P c878fd758eff4fc5a0798609b8c74bd1: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260926 11:33:47.949326 18442 ts_tablet_manager.cc:1428] T 7c1c551449ed4512a7a76e4247193921 P 57995f568fe8413896c2592d3b329be2: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260926 11:33:47.949339 18441 ts_tablet_manager.cc:1428] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 11:33:47.949668 18161 heartbeater.cc:503] Master 127.0.98.62:40687 was elected leader, sending a full tablet report...
I20260926 11:33:47.949668 18423 heartbeater.cc:503] Master 127.0.98.62:40687 was elected leader, sending a full tablet report...
I20260926 11:33:47.950639 18292 heartbeater.cc:503] Master 127.0.98.62:40687 was elected leader, sending a full tablet report...
I20260926 11:33:47.951494 18446 raft_consensus.cc:482] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:33:47.951646 18446 raft_consensus.cc:504] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57995f568fe8413896c2592d3b329be2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44671 } } peers { permanent_uuid: "c878fd758eff4fc5a0798609b8c74bd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43755 } } peers { permanent_uuid: "8436329f0c7c4ab9a0b2afeb7ff3c8c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46065 } }
I20260926 11:33:47.953630 18446 leader_election.cc:288] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57995f568fe8413896c2592d3b329be2 (127.0.98.1:44671), c878fd758eff4fc5a0798609b8c74bd1 (127.0.98.2:43755)
I20260926 11:33:47.957257 18425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:47.953036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44671 (local address 127.0.98.3:60921)
0926 11:33:47.953137 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:47.953153 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:47.953232 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:47.953765 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:47.953771 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:47.953790 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:47.953912 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:47.953921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.954850 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.954854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:47.956043 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:47.956054 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.956835 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.956838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:47.956943 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:47.956965 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 11:33:47.957059 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 11:33:47.957132 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 11:33:47.957257 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:47.953293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60921 (local address 127.0.98.1:44671)
0926 11:33:47.953457 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:47.953461 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:47.953472 (+ 11us) server_negotiation.cc:407] Connection header received
0926 11:33:47.953507 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:47.953509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:47.953572 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:47.953649 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:47.954004 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.954711 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.956161 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.956721 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.956832 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:47.956889 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:47.957084 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 11:33:47.957148 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 11:33:47.957867 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:47.953597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54097 (local address 127.0.98.2:43755)
0926 11:33:47.953703 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:47.953709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:47.953793 (+ 84us) server_negotiation.cc:407] Connection header received
0926 11:33:47.953973 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:47.953976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:47.954021 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:47.954095 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:47.954521 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.955127 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.956644 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.957165 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.957268 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:47.957334 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:47.957672 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 11:33:47.957746 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 11:33:47.957863 18115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c1c551449ed4512a7a76e4247193921" candidate_uuid: "8436329f0c7c4ab9a0b2afeb7ff3c8c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57995f568fe8413896c2592d3b329be2" is_pre_election: true
I20260926 11:33:47.958163 18115 raft_consensus.cc:2454] T 7c1c551449ed4512a7a76e4247193921 P 57995f568fe8413896c2592d3b329be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8436329f0c7c4ab9a0b2afeb7ff3c8c5 in term 0.
I20260926 11:33:47.958211 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:47.953399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43755 (local address 127.0.98.3:54097)
0926 11:33:47.953715 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:47.953731 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:47.953843 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:47.954252 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:47.954255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:47.954273 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:47.954397 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:47.954403 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.955230 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.955235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:47.956492 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:47.956502 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.957408 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.957414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:47.957529 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:47.957552 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 11:33:47.958032 (+ 480us) client_negotiation.cc:240] Negotiation successful
0926 11:33:47.958110 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":106,"threads_started":1}
I20260926 11:33:47.958658 18311 leader_election.cc:302] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57995f568fe8413896c2592d3b329be2, 8436329f0c7c4ab9a0b2afeb7ff3c8c5; no voters:
I20260926 11:33:47.958856 18246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c1c551449ed4512a7a76e4247193921" candidate_uuid: "8436329f0c7c4ab9a0b2afeb7ff3c8c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c878fd758eff4fc5a0798609b8c74bd1" is_pre_election: true
I20260926 11:33:47.958936 18446 raft_consensus.cc:2781] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:33:47.959021 18446 raft_consensus.cc:482] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:33:47.959072 18446 raft_consensus.cc:3037] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:33:47.959123 18246 raft_consensus.cc:2454] T 7c1c551449ed4512a7a76e4247193921 P c878fd758eff4fc5a0798609b8c74bd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8436329f0c7c4ab9a0b2afeb7ff3c8c5 in term 0.
I20260926 11:33:47.960258 18446 raft_consensus.cc:504] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57995f568fe8413896c2592d3b329be2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44671 } } peers { permanent_uuid: "c878fd758eff4fc5a0798609b8c74bd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43755 } } peers { permanent_uuid: "8436329f0c7c4ab9a0b2afeb7ff3c8c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46065 } }
I20260926 11:33:47.960618 18446 leader_election.cc:288] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5 [CANDIDATE]: Term 1 election: Requested vote from peers 57995f568fe8413896c2592d3b329be2 (127.0.98.1:44671), c878fd758eff4fc5a0798609b8c74bd1 (127.0.98.2:43755)
I20260926 11:33:47.960913 18115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c1c551449ed4512a7a76e4247193921" candidate_uuid: "8436329f0c7c4ab9a0b2afeb7ff3c8c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57995f568fe8413896c2592d3b329be2"
I20260926 11:33:47.960963 18246 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c1c551449ed4512a7a76e4247193921" candidate_uuid: "8436329f0c7c4ab9a0b2afeb7ff3c8c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c878fd758eff4fc5a0798609b8c74bd1"
I20260926 11:33:47.961024 18115 raft_consensus.cc:3037] T 7c1c551449ed4512a7a76e4247193921 P 57995f568fe8413896c2592d3b329be2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:33:47.961097 18246 raft_consensus.cc:3037] T 7c1c551449ed4512a7a76e4247193921 P c878fd758eff4fc5a0798609b8c74bd1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:33:47.962031 18115 raft_consensus.cc:2454] T 7c1c551449ed4512a7a76e4247193921 P 57995f568fe8413896c2592d3b329be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8436329f0c7c4ab9a0b2afeb7ff3c8c5 in term 1.
I20260926 11:33:47.962147 18246 raft_consensus.cc:2454] T 7c1c551449ed4512a7a76e4247193921 P c878fd758eff4fc5a0798609b8c74bd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8436329f0c7c4ab9a0b2afeb7ff3c8c5 in term 1.
I20260926 11:33:47.962401 18311 leader_election.cc:302] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 57995f568fe8413896c2592d3b329be2, 8436329f0c7c4ab9a0b2afeb7ff3c8c5; no voters:
I20260926 11:33:47.962556 18446 raft_consensus.cc:2781] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:33:47.962766 18446 raft_consensus.cc:686] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5 [term 1 LEADER]: Becoming Leader. State: Replica: 8436329f0c7c4ab9a0b2afeb7ff3c8c5, State: Running, Role: LEADER
I20260926 11:33:47.963076 18446 consensus_queue.cc:237] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57995f568fe8413896c2592d3b329be2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44671 } } peers { permanent_uuid: "c878fd758eff4fc5a0798609b8c74bd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43755 } } peers { permanent_uuid: "8436329f0c7c4ab9a0b2afeb7ff3c8c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46065 } }
I20260926 11:33:47.966212 17973 catalog_manager.cc:5495] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8436329f0c7c4ab9a0b2afeb7ff3c8c5 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8436329f0c7c4ab9a0b2afeb7ff3c8c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57995f568fe8413896c2592d3b329be2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c878fd758eff4fc5a0798609b8c74bd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8436329f0c7c4ab9a0b2afeb7ff3c8c5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46065 } health_report { overall_health: HEALTHY } } }
I20260926 11:33:47.989049 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:47.985454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53832 (local address 127.0.98.3:46065)
0926 11:33:47.985590 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:47.985595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:47.985610 (+ 15us) server_negotiation.cc:407] Connection header received
0926 11:33:47.985721 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:47.985723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:47.985776 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:47.985890 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:33:47.986135 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.986670 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.987556 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:47.987803 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:47.987916 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:47.988227 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:33:47.988659 (+ 432us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:33:47.988806 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:47.988846 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 11:33:47.988897 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 11:33:47.994841 18246 raft_consensus.cc:1261] T 7c1c551449ed4512a7a76e4247193921 P c878fd758eff4fc5a0798609b8c74bd1 [term 1 FOLLOWER]: Refusing update from remote peer 8436329f0c7c4ab9a0b2afeb7ff3c8c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:33:47.994841 18115 raft_consensus.cc:1261] T 7c1c551449ed4512a7a76e4247193921 P 57995f568fe8413896c2592d3b329be2 [term 1 FOLLOWER]: Refusing update from remote peer 8436329f0c7c4ab9a0b2afeb7ff3c8c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:33:47.995580 18446 consensus_queue.cc:1035] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c878fd758eff4fc5a0798609b8c74bd1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:33:47.995811 18450 consensus_queue.cc:1035] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "57995f568fe8413896c2592d3b329be2" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:33:48.003677 18455 mvcc.cc:205] Tried to move back new op lower bound from 7333570265059962880 to 7333570264939507712. Current Snapshot: MvccSnapshot[applied={T|T < 7333570265059962880}]
I20260926 11:33:48.012764 17973 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:33:48.012951 17973 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:33:48.015432 17973 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3b7d7f52a9db4aa2952797ad27c60d88: Sending DeleteTablet for 3 replicas of tablet 7c1c551449ed4512a7a76e4247193921
I20260926 11:33:48.016395 18226 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c1c551449ed4512a7a76e4247193921 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:33:48 UTC) from {username='slave'} at 127.0.0.1:55588
I20260926 11:33:48.016350 18357 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c1c551449ed4512a7a76e4247193921 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:33:48 UTC) from {username='slave'} at 127.0.0.1:53826
I20260926 11:33:48.016431 18095 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c1c551449ed4512a7a76e4247193921 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:33:48 UTC) from {username='slave'} at 127.0.0.1:34038
I20260926 11:33:48.016891 18463 tablet_replica.cc:330] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5: stopping tablet replica
I20260926 11:33:48.016939 18465 tablet_replica.cc:330] T 7c1c551449ed4512a7a76e4247193921 P 57995f568fe8413896c2592d3b329be2: stopping tablet replica
I20260926 11:33:48.017262 18465 raft_consensus.cc:2229] T 7c1c551449ed4512a7a76e4247193921 P 57995f568fe8413896c2592d3b329be2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:33:48.017272 18463 raft_consensus.cc:2229] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:33:48.017431 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 18033
I20260926 11:33:48.018362 18463 raft_consensus.cc:2258] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:33:48.019800 18463 ts_tablet_manager.cc:1905] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:33:48.022698 18463 ts_tablet_manager.cc:1918] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:33:48.022804 18463 log.cc:1196] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790422180210734-392-0/minicluster-data/ts-2/wal/wals/7c1c551449ed4512a7a76e4247193921
I20260926 11:33:48.023128 18463 ts_tablet_manager.cc:1939] T 7c1c551449ed4512a7a76e4247193921 P 8436329f0c7c4ab9a0b2afeb7ff3c8c5: Deleting consensus metadata
I20260926 11:33:48.023140 18464 tablet_replica.cc:330] T 7c1c551449ed4512a7a76e4247193921 P c878fd758eff4fc5a0798609b8c74bd1: stopping tablet replica
I20260926 11:33:48.023635 18464 raft_consensus.cc:2229] T 7c1c551449ed4512a7a76e4247193921 P c878fd758eff4fc5a0798609b8c74bd1 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 11:33:48.023655 17958 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44671 (error 108)
I20260926 11:33:48.023797 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 18164
I20260926 11:33:48.024225 18464 raft_consensus.cc:2258] T 7c1c551449ed4512a7a76e4247193921 P c878fd758eff4fc5a0798609b8c74bd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:33:48.025969 17960 catalog_manager.cc:4841] TS 8436329f0c7c4ab9a0b2afeb7ff3c8c5 (127.0.98.3:46065): tablet 7c1c551449ed4512a7a76e4247193921 (table test-table [id=d34fe65777e54887999f66d7fe9fbd5c]) successfully deleted
I20260926 11:33:48.030645 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 18295
I20260926 11:33:48.032362 18436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:48.030914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44671 (local address 127.0.0.1:34044)
0926 11:33:48.031642 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:48.032231 (+ 589us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":615}
I20260926 11:33:48.032549 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:48.031144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43755 (local address 127.0.0.1:55604)
0926 11:33:48.032383 (+ 1239us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:48.032442 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43755: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1152}
W20260926 11:33:48.032593 17958 catalog_manager.cc:4584] TS 57995f568fe8413896c2592d3b329be2 (127.0.98.1:44671): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7c1c551449ed4512a7a76e4247193921: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44671: connect: Connection refused (error 111)
I20260926 11:33:48.036777 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 17942
2026-09-26T11:33:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1003 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:33:48 dist-test-slave-mrck krb5kdc[18469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:33:48 dist-test-slave-mrck krb5kdc[18469](info): set up 2 sockets
Sep 26 11:33:48 dist-test-slave-mrck krb5kdc[18469](info): commencing operation
krb5kdc: starting...
W20260926 11:33:49.095607 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:33:49 dist-test-slave-mrck krb5kdc[18469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:33:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:33:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:33:49.214967 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36313
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:33:49.329232 18482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:49.329550 18482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:49.329593 18482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:49.333849 18482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:33:49.333923 18482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:49.333949 18482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:49.333967 18482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:33:49.334012 18482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:33:49.339514 18482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:49.340719 18482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:49.341823 18482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:33:49.350471 18488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:49.350569 18490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:49.350562 18487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:49.350631 18482 server_base.cc:1034] running on GCE node
I20260926 11:33:49.351408 18482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:49.352655 18482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:49.353886 18482 hybrid_clock.cc:648] HybridClock initialized: now 1790422429353865 us; error 36 us; skew 500 ppm
Sep 26 11:33:49 dist-test-slave-mrck krb5kdc[18469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:33:49.356827 18482 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:33:49.357733 18482 webserver.cc:460] Webserver started at http://127.0.98.62:44731/ using document root <none> and password file <none>
I20260926 11:33:49.358318 18482 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:49.358382 18482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:49.358610 18482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:33:49.360865 18482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "00296ffe499e41b096bc5fafbbf85f16"
format_stamp: "Formatted at 2026-09-26 11:33:49 on dist-test-slave-mrck"
server_key: "68152f6434b98a31dfbc844ef31b668c"
server_key_iv: "0d3e77ead0a3339dc6a6b3a104ddbddb"
server_key_version: "encryptionkey@0"
I20260926 11:33:49.361461 18482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "00296ffe499e41b096bc5fafbbf85f16"
format_stamp: "Formatted at 2026-09-26 11:33:49 on dist-test-slave-mrck"
server_key: "68152f6434b98a31dfbc844ef31b668c"
server_key_iv: "0d3e77ead0a3339dc6a6b3a104ddbddb"
server_key_version: "encryptionkey@0"
I20260926 11:33:49.365262 18482 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:33:49.367846 18497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:49.369076 18482 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:33:49.369179 18482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "00296ffe499e41b096bc5fafbbf85f16"
format_stamp: "Formatted at 2026-09-26 11:33:49 on dist-test-slave-mrck"
server_key: "68152f6434b98a31dfbc844ef31b668c"
server_key_iv: "0d3e77ead0a3339dc6a6b3a104ddbddb"
server_key_version: "encryptionkey@0"
I20260926 11:33:49.369320 18482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:49.408396 18482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:49.411909 18482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:49.412122 18482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:49.419018 18482 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36313
I20260926 11:33:49.419080 18549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36313 every 8 connection(s)
I20260926 11:33:49.420194 18482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:33:49.421705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 18482
I20260926 11:33:49.421852 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:33:49.422178 392 external_mini_cluster.cc:1428] Setting key 423f054e1e93a01bf596ae64d9314ca6
I20260926 11:33:49.423400 18550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:33:49.428906 18550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00296ffe499e41b096bc5fafbbf85f16: Bootstrap starting.
Sep 26 11:33:49 dist-test-slave-mrck krb5kdc[18469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:33:49.431146 18550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00296ffe499e41b096bc5fafbbf85f16: Neither blocks nor log segments found. Creating new log.
I20260926 11:33:49.431954 18550 log.cc:824] T 00000000000000000000000000000000 P 00296ffe499e41b096bc5fafbbf85f16: Log is configured to *not* fsync() on all Append() calls
I20260926 11:33:49.433972 18550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00296ffe499e41b096bc5fafbbf85f16: No bootstrap required, opened a new log
I20260926 11:33:49.436419 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:49.423675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50324 (local address 127.0.98.62:36313)
0926 11:33:49.424158 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:49.424173 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:49.424208 (+ 35us) server_negotiation.cc:407] Connection header received
0926 11:33:49.424900 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:49.424918 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:49.425249 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:49.425600 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:49.426423 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:49.427418 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:49.428355 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:49.428829 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:49.429014 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:49.431547 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:49.431552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:49.431565 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:33:49.431591 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:49.433470 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:49.433935 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:49.433941 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:49.433947 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:49.434114 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:49.434428 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:49.434431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:49.434433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:49.434826 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:49.435053 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:49.435334 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 11:33:49.435638 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":145,"threads_started":1}
I20260926 11:33:49.437879 18550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00296ffe499e41b096bc5fafbbf85f16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00296ffe499e41b096bc5fafbbf85f16" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36313 } }
I20260926 11:33:49.438143 18550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00296ffe499e41b096bc5fafbbf85f16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:33:49.438190 18550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00296ffe499e41b096bc5fafbbf85f16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00296ffe499e41b096bc5fafbbf85f16, State: Initialized, Role: FOLLOWER
I20260926 11:33:49.438653 18550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00296ffe499e41b096bc5fafbbf85f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00296ffe499e41b096bc5fafbbf85f16" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36313 } }
I20260926 11:33:49.438813 18550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00296ffe499e41b096bc5fafbbf85f16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:33:49.438886 18550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00296ffe499e41b096bc5fafbbf85f16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:33:49.438974 18550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00296ffe499e41b096bc5fafbbf85f16 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:33:49.439867 18550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00296ffe499e41b096bc5fafbbf85f16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00296ffe499e41b096bc5fafbbf85f16" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36313 } }
I20260926 11:33:49.440155 18550 leader_election.cc:302] T 00000000000000000000000000000000 P 00296ffe499e41b096bc5fafbbf85f16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00296ffe499e41b096bc5fafbbf85f16; no voters:
I20260926 11:33:49.440398 18550 leader_election.cc:288] T 00000000000000000000000000000000 P 00296ffe499e41b096bc5fafbbf85f16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:33:49.440521 18555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00296ffe499e41b096bc5fafbbf85f16 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:33:49.440798 18555 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00296ffe499e41b096bc5fafbbf85f16 [term 1 LEADER]: Becoming Leader. State: Replica: 00296ffe499e41b096bc5fafbbf85f16, State: Running, Role: LEADER
I20260926 11:33:49.441228 18555 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00296ffe499e41b096bc5fafbbf85f16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00296ffe499e41b096bc5fafbbf85f16" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36313 } }
I20260926 11:33:49.441597 18550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00296ffe499e41b096bc5fafbbf85f16 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:33:49.443130 18557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00296ffe499e41b096bc5fafbbf85f16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00296ffe499e41b096bc5fafbbf85f16. Latest consensus state: current_term: 1 leader_uuid: "00296ffe499e41b096bc5fafbbf85f16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00296ffe499e41b096bc5fafbbf85f16" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36313 } } }
I20260926 11:33:49.443122 18556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00296ffe499e41b096bc5fafbbf85f16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00296ffe499e41b096bc5fafbbf85f16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00296ffe499e41b096bc5fafbbf85f16" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36313 } } }
I20260926 11:33:49.443290 18557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00296ffe499e41b096bc5fafbbf85f16 [sys.catalog]: This master's current role is: LEADER
I20260926 11:33:49.443290 18556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00296ffe499e41b096bc5fafbbf85f16 [sys.catalog]: This master's current role is: LEADER
I20260926 11:33:49.443671 18564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:33:49.446408 18564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:33:49.451871 18564 catalog_manager.cc:1347] Generated new cluster ID: 6fda52ace0154c6289aa6cd620f31502
I20260926 11:33:49.451939 18564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:33:49.475796 18564 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:33:49.477095 18564 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:33:49.482959 18564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00296ffe499e41b096bc5fafbbf85f16: Generated new TSK 0
I20260926 11:33:49.483773 18564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:33:49.510929 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36313
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:33:49.626683 18576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:49.626981 18576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:49.627063 18576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:49.631117 18576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:49.631207 18576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:49.631294 18576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:33:49.636632 18576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:49.637770 18576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:49.638799 18576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:33:49.648598 18584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:49.648676 18576 server_base.cc:1034] running on GCE node
W20260926 11:33:49.648591 18581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:49.648598 18582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:49.649338 18576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:49.649984 18576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:49.651163 18576 hybrid_clock.cc:648] HybridClock initialized: now 1790422429651148 us; error 27 us; skew 500 ppm
Sep 26 11:33:49 dist-test-slave-mrck krb5kdc[18469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:33:49.653947 18576 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:33:49.654816 18576 webserver.cc:460] Webserver started at http://127.0.98.1:45537/ using document root <none> and password file <none>
I20260926 11:33:49.655395 18576 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:49.655505 18576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:49.655773 18576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:33:49.657898 18576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-0/data/instance:
uuid: "1f63de4907154edfae24c3f8275496b0"
format_stamp: "Formatted at 2026-09-26 11:33:49 on dist-test-slave-mrck"
server_key: "b1e0d3a811dddac9b9fa407809f10f27"
server_key_iv: "e80925fc557146fceb98777712deffc7"
server_key_version: "encryptionkey@0"
I20260926 11:33:49.658481 18576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1f63de4907154edfae24c3f8275496b0"
format_stamp: "Formatted at 2026-09-26 11:33:49 on dist-test-slave-mrck"
server_key: "b1e0d3a811dddac9b9fa407809f10f27"
server_key_iv: "e80925fc557146fceb98777712deffc7"
server_key_version: "encryptionkey@0"
I20260926 11:33:49.662166 18576 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 11:33:49.664623 18591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:49.665791 18576 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260926 11:33:49.665917 18576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "1f63de4907154edfae24c3f8275496b0"
format_stamp: "Formatted at 2026-09-26 11:33:49 on dist-test-slave-mrck"
server_key: "b1e0d3a811dddac9b9fa407809f10f27"
server_key_iv: "e80925fc557146fceb98777712deffc7"
server_key_version: "encryptionkey@0"
I20260926 11:33:49.666026 18576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:49.677949 18576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:49.681265 18576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:49.681481 18576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:49.682063 18576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:33:49.682930 18576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:33:49.683000 18576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:49.683073 18576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:33:49.683117 18576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:49.692147 18576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33581
I20260926 11:33:49.692169 18704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33581 every 8 connection(s)
I20260926 11:33:49.693048 18576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:33:49.697475 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 18576
I20260926 11:33:49.697616 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:33:49.697906 392 external_mini_cluster.cc:1428] Setting key 9bcaf9823bf7f0e393d06a5223db250d
Sep 26 11:33:49 dist-test-slave-mrck krb5kdc[18469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:33:49.707641 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:49.694857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60317 (local address 127.0.98.62:36313)
0926 11:33:49.695004 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:49.695007 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:49.695802 (+ 795us) server_negotiation.cc:407] Connection header received
0926 11:33:49.696715 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:49.696718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:49.696774 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:49.696879 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:49.698169 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:49.698888 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:49.700658 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:49.700943 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:49.701074 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:49.703786 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:49.703793 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:49.703799 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:33:49.703835 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:49.705536 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:49.706032 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:49.706037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:49.706041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:49.706134 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:49.706450 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:49.706454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:49.706457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:49.706716 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:49.706828 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:49.707351 (+ 523us) server_negotiation.cc:299] Negotiation successful
0926 11:33:49.707512 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 11:33:49.708984 18707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:49.695222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36313 (local address 127.0.98.1:60317)
0926 11:33:49.695640 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:49.695680 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:49.696500 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:49.697057 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:49.697064 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:49.697421 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:33:49.698019 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:49.698044 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:49.699046 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:49.699050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:49.700529 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:49.700544 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:49.701122 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:49.701126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:49.701278 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:49.701870 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:33:49.701886 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:33:49.703578 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:33:49.705668 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:33:49.705673 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:33:49.705685 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:33:49.705930 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:33:49.706231 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:33:49.706233 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:33:49.706234 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:33:49.706357 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:33:49.706816 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:33:49.706821 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:33:49.707066 (+ 245us) client_negotiation.cc:769] Sending connection context
0926 11:33:49.707853 (+ 787us) client_negotiation.cc:240] Negotiation successful
0926 11:33:49.708160 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 11:33:49.710218 18705 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36313
I20260926 11:33:49.710476 18705 heartbeater.cc:464] Registering TS with master...
I20260926 11:33:49.710965 18705 heartbeater.cc:511] Master 127.0.98.62:36313 requested a full tablet report, sending...
I20260926 11:33:49.712633 18509 ts_manager.cc:194] Registered new tserver with Master: 1f63de4907154edfae24c3f8275496b0 (127.0.98.1:33581)
I20260926 11:33:49.713868 18509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60317
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:33:49.724733 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36313
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:33:49.833413 18710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:49.833657 18710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:49.833698 18710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:49.837630 18710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:49.837695 18710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:49.837791 18710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:33:49.843106 18710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:49.844202 18710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:49.845286 18710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:33:49.855134 18718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:49.855239 18710 server_base.cc:1034] running on GCE node
W20260926 11:33:49.855156 18716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:49.855162 18715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:49.855886 18710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:49.856498 18710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:49.857681 18710 hybrid_clock.cc:648] HybridClock initialized: now 1790422429857661 us; error 29 us; skew 500 ppm
Sep 26 11:33:49 dist-test-slave-mrck krb5kdc[18469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:33:49.860561 18710 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:33:49.861445 18710 webserver.cc:460] Webserver started at http://127.0.98.2:45935/ using document root <none> and password file <none>
I20260926 11:33:49.862006 18710 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:49.862097 18710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:49.862325 18710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:33:49.864398 18710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-1/data/instance:
uuid: "9f9ec79fcd264daa85b7cc62ceb9ac17"
format_stamp: "Formatted at 2026-09-26 11:33:49 on dist-test-slave-mrck"
server_key: "b4ca7af7392b4e31801c141a1d970168"
server_key_iv: "b3bbf67a21e76ca522f72dacf926e4dc"
server_key_version: "encryptionkey@0"
I20260926 11:33:49.864966 18710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9f9ec79fcd264daa85b7cc62ceb9ac17"
format_stamp: "Formatted at 2026-09-26 11:33:49 on dist-test-slave-mrck"
server_key: "b4ca7af7392b4e31801c141a1d970168"
server_key_iv: "b3bbf67a21e76ca522f72dacf926e4dc"
server_key_version: "encryptionkey@0"
I20260926 11:33:49.868615 18710 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:33:49.871138 18725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:49.872393 18710 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:33:49.872534 18710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "9f9ec79fcd264daa85b7cc62ceb9ac17"
format_stamp: "Formatted at 2026-09-26 11:33:49 on dist-test-slave-mrck"
server_key: "b4ca7af7392b4e31801c141a1d970168"
server_key_iv: "b3bbf67a21e76ca522f72dacf926e4dc"
server_key_version: "encryptionkey@0"
I20260926 11:33:49.872644 18710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:49.910879 18710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:49.913954 18710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:49.914191 18710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:49.914790 18710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:33:49.915612 18710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:33:49.915681 18710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:49.915768 18710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:33:49.915838 18710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:49.925292 18710 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40747
I20260926 11:33:49.925308 18838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40747 every 8 connection(s)
I20260926 11:33:49.926194 18710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:33:49.931385 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 18710
I20260926 11:33:49.931528 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:33:49.931847 392 external_mini_cluster.cc:1428] Setting key 9ee050dd1301641baa363e3037bd2b42
Sep 26 11:33:49 dist-test-slave-mrck krb5kdc[18469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:33:49.940186 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:49.927984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35171 (local address 127.0.98.62:36313)
0926 11:33:49.928156 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:49.928160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:49.929020 (+ 860us) server_negotiation.cc:407] Connection header received
0926 11:33:49.929858 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:49.929861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:49.929908 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:49.929987 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:49.931176 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:49.932014 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:49.933245 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:49.933487 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:49.933602 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:49.936342 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:49.936349 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:49.936355 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:33:49.936387 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:49.938068 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:49.938545 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:49.938551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:49.938556 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:49.938650 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:49.938989 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:49.938996 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:49.938999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:49.939264 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:49.939386 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:49.939873 (+ 487us) server_negotiation.cc:299] Negotiation successful
0926 11:33:49.939998 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 11:33:49.940959 18841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:49.928412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36313 (local address 127.0.98.2:35171)
0926 11:33:49.928858 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:49.928900 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:49.929664 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:49.930155 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:49.930163 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:49.930528 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:33:49.931006 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:49.931019 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:49.932147 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:49.932150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:49.933123 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:49.933132 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:49.933636 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:49.933640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:49.933822 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:49.934451 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:33:49.934474 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:33:49.936176 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:33:49.938180 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:33:49.938184 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:33:49.938196 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:33:49.938443 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:33:49.938749 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:33:49.938751 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:33:49.938753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:33:49.938877 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:33:49.939388 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:33:49.939392 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:33:49.939610 (+ 218us) client_negotiation.cc:769] Sending connection context
0926 11:33:49.939823 (+ 213us) client_negotiation.cc:240] Negotiation successful
0926 11:33:49.940086 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 11:33:49.942255 18839 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36313
I20260926 11:33:49.942536 18839 heartbeater.cc:464] Registering TS with master...
I20260926 11:33:49.943025 18839 heartbeater.cc:511] Master 127.0.98.62:36313 requested a full tablet report, sending...
I20260926 11:33:49.944134 18509 ts_manager.cc:194] Registered new tserver with Master: 9f9ec79fcd264daa85b7cc62ceb9ac17 (127.0.98.2:40747)
I20260926 11:33:49.944756 18509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:33:49.958611 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36313
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:33:50.068491 18844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:50.068753 18844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:50.068796 18844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:50.073073 18844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:50.073148 18844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:50.073236 18844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:33:50.078862 18844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:50.080073 18844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:50.081106 18844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 11:33:50.091290 18844 server_base.cc:1034] running on GCE node
W20260926 11:33:50.091266 18850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:50.091266 18852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:50.091281 18849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:50.092087 18844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:50.092708 18844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:50.093866 18844 hybrid_clock.cc:648] HybridClock initialized: now 1790422430093853 us; error 30 us; skew 500 ppm
Sep 26 11:33:50 dist-test-slave-mrck krb5kdc[18469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:33:50.096590 18844 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:33:50.097492 18844 webserver.cc:460] Webserver started at http://127.0.98.3:41629/ using document root <none> and password file <none>
I20260926 11:33:50.098059 18844 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:50.098150 18844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:50.098377 18844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:33:50.100442 18844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-2/data/instance:
uuid: "31a855c7ddb44bacb6dbf049fb2288b0"
format_stamp: "Formatted at 2026-09-26 11:33:50 on dist-test-slave-mrck"
server_key: "9fc4e34857b0a120993f6f586a387896"
server_key_iv: "a721c0a5585a6fbb1796c20bdab5c8d4"
server_key_version: "encryptionkey@0"
I20260926 11:33:50.101003 18844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "31a855c7ddb44bacb6dbf049fb2288b0"
format_stamp: "Formatted at 2026-09-26 11:33:50 on dist-test-slave-mrck"
server_key: "9fc4e34857b0a120993f6f586a387896"
server_key_iv: "a721c0a5585a6fbb1796c20bdab5c8d4"
server_key_version: "encryptionkey@0"
I20260926 11:33:50.104733 18844 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:33:50.107136 18859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:50.108516 18844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 11:33:50.108649 18844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "31a855c7ddb44bacb6dbf049fb2288b0"
format_stamp: "Formatted at 2026-09-26 11:33:50 on dist-test-slave-mrck"
server_key: "9fc4e34857b0a120993f6f586a387896"
server_key_iv: "a721c0a5585a6fbb1796c20bdab5c8d4"
server_key_version: "encryptionkey@0"
I20260926 11:33:50.108769 18844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:50.132162 18844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:50.135282 18844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:50.135610 18844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:50.136292 18844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:33:50.137142 18844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:33:50.137212 18844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:50.137297 18844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:33:50.137339 18844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:50.146588 18844 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35265
I20260926 11:33:50.146615 18972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35265 every 8 connection(s)
I20260926 11:33:50.147507 18844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:33:50.155134 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 18844
I20260926 11:33:50.155298 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:33:50.155637 392 external_mini_cluster.cc:1428] Setting key b5eec9627d9a8b0ab3154572401252bc
Sep 26 11:33:50 dist-test-slave-mrck krb5kdc[18469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422430, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:33:50.161134 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:50.149373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45957 (local address 127.0.98.62:36313)
0926 11:33:50.149521 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:50.149526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:50.150407 (+ 881us) server_negotiation.cc:407] Connection header received
0926 11:33:50.151251 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:50.151253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:50.151299 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:50.151383 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:50.152603 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.153113 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.154182 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.154398 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.154472 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:50.157393 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:50.157397 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:50.157400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:33:50.157428 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:50.159069 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:50.159569 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:50.159572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:50.159573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:50.159645 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:50.160106 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:50.160109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:50.160110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:50.160285 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:50.160359 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:50.160877 (+ 518us) server_negotiation.cc:299] Negotiation successful
0926 11:33:50.160999 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 11:33:50.161906 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:50.149818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36313 (local address 127.0.98.3:45957)
0926 11:33:50.150252 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:50.150293 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:50.151062 (+ 769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:50.151540 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:50.151546 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:50.151990 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:33:50.152468 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:50.152481 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.153216 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.153219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:50.154079 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:50.154088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.154516 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.154518 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:50.154687 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:50.155335 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:33:50.155358 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:33:50.157243 (+ 1885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:33:50.159191 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:33:50.159196 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:33:50.159208 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:33:50.159483 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:33:50.159841 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:33:50.159844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:33:50.159846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:33:50.160012 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:33:50.160383 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:33:50.160389 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:33:50.160626 (+ 237us) client_negotiation.cc:769] Sending connection context
0926 11:33:50.160786 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 11:33:50.161074 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":104,"threads_started":1}
I20260926 11:33:50.162998 18973 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36313
I20260926 11:33:50.163203 18973 heartbeater.cc:464] Registering TS with master...
I20260926 11:33:50.163650 18973 heartbeater.cc:511] Master 127.0.98.62:36313 requested a full tablet report, sending...
I20260926 11:33:50.164685 18509 ts_manager.cc:194] Registered new tserver with Master: 31a855c7ddb44bacb6dbf049fb2288b0 (127.0.98.3:35265)
I20260926 11:33:50.165226 18509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45957
I20260926 11:33:50.169596 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:33:50.179467 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:50.171511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50336 (local address 127.0.98.62:36313)
0926 11:33:50.171654 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:50.171658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:50.171893 (+ 235us) server_negotiation.cc:407] Connection header received
0926 11:33:50.172061 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:50.172064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:50.172113 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:50.172220 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:50.173012 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.173524 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.174400 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.174618 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.174721 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:50.176321 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:50.176325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:50.176328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:33:50.176361 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:50.177879 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:50.178398 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:50.178400 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:50.178401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:50.178470 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:50.178744 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:50.178747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:50.178748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:50.178930 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:50.179046 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:50.179282 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 11:33:50.179354 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 11:33:50.182335 18509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50336:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:33:50.185524 18509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:33:50.200299 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:50.194042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40747 (local address 127.0.0.1:33580)
0926 11:33:50.194440 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:50.194486 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:50.194607 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:50.196085 (+ 1478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:50.196089 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:50.196107 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:50.196195 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:50.196200 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.197666 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.197672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:50.198733 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:50.198744 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.199812 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.199816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:50.199919 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:50.199941 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 11:33:50.199993 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 11:33:50.200082 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260926 11:33:50.200294 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:50.194073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33581 (local address 127.0.0.1:41716)
0926 11:33:50.194705 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:50.194718 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:50.194808 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:50.195668 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:50.195675 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:50.195698 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:50.195880 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:50.195887 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.197481 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.197485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:50.198633 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:50.198643 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.199812 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.199816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:50.199969 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:50.199999 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 11:33:50.200084 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 11:33:50.200141 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":68,"thread_start_us":112,"threads_started":1}
I20260926 11:33:50.200379 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:50.195157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39770 (local address 127.0.98.3:35265)
0926 11:33:50.195527 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:50.195539 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:50.195559 (+ 20us) server_negotiation.cc:407] Connection header received
0926 11:33:50.195635 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:50.195642 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:50.195855 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:50.196031 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:50.196239 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.196978 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.198766 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.199473 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.199621 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:50.199795 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:50.200123 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 11:33:50.200231 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":126,"threads_started":1}
I20260926 11:33:50.200629 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:50.194837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41716 (local address 127.0.98.1:33581)
0926 11:33:50.195129 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:50.195144 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:50.195166 (+ 22us) server_negotiation.cc:407] Connection header received
0926 11:33:50.195245 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:50.195252 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:50.195478 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:50.196038 (+ 560us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:50.196198 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.197348 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.198820 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.199643 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.199863 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:50.200020 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:50.200363 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 11:33:50.200478 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":74,"threads_started":1}
I20260926 11:33:50.200294 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:50.194920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35265 (local address 127.0.0.1:39770)
0926 11:33:50.195195 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:50.195208 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:50.195288 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:50.196003 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:50.196006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:50.196023 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:50.196121 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:50.196128 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.197101 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.197107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:50.198633 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:50.198643 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.199638 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.199642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:50.199777 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:50.199822 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 11:33:50.199940 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 11:33:50.200078 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":63,"thread_start_us":86,"threads_started":1}
I20260926 11:33:50.201116 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:50.194188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33580 (local address 127.0.98.2:40747)
0926 11:33:50.195609 (+ 1421us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:50.195622 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:50.195645 (+ 23us) server_negotiation.cc:407] Connection header received
0926 11:33:50.195713 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:50.195717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:50.195939 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:50.196330 (+ 391us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:50.196493 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.197544 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.198850 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.199643 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.200677 (+ 1034us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:50.200810 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:50.200857 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 11:33:50.200954 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1301,"thread_start_us":123,"threads_started":1}
I20260926 11:33:50.203626 18773 tablet_service.cc:1467] Processing CreateTablet for tablet 71b5c7a4316548ab96131db94a1cd0d1 (DEFAULT_TABLE table=test-table [id=696260f39ed04b268a66efc9a9573ea7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:33:50.203675 18639 tablet_service.cc:1467] Processing CreateTablet for tablet 71b5c7a4316548ab96131db94a1cd0d1 (DEFAULT_TABLE table=test-table [id=696260f39ed04b268a66efc9a9573ea7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:33:50.203671 18907 tablet_service.cc:1467] Processing CreateTablet for tablet 71b5c7a4316548ab96131db94a1cd0d1 (DEFAULT_TABLE table=test-table [id=696260f39ed04b268a66efc9a9573ea7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:33:50.204698 18773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71b5c7a4316548ab96131db94a1cd0d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:33:50.204713 18639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71b5c7a4316548ab96131db94a1cd0d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:33:50.204713 18907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71b5c7a4316548ab96131db94a1cd0d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:33:50.210979 18990 tablet_bootstrap.cc:492] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17: Bootstrap starting.
I20260926 11:33:50.211796 18991 tablet_bootstrap.cc:492] T 71b5c7a4316548ab96131db94a1cd0d1 P 1f63de4907154edfae24c3f8275496b0: Bootstrap starting.
I20260926 11:33:50.212639 18992 tablet_bootstrap.cc:492] T 71b5c7a4316548ab96131db94a1cd0d1 P 31a855c7ddb44bacb6dbf049fb2288b0: Bootstrap starting.
I20260926 11:33:50.213467 18990 tablet_bootstrap.cc:654] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17: Neither blocks nor log segments found. Creating new log.
I20260926 11:33:50.213899 18991 tablet_bootstrap.cc:654] T 71b5c7a4316548ab96131db94a1cd0d1 P 1f63de4907154edfae24c3f8275496b0: Neither blocks nor log segments found. Creating new log.
I20260926 11:33:50.214509 18992 tablet_bootstrap.cc:654] T 71b5c7a4316548ab96131db94a1cd0d1 P 31a855c7ddb44bacb6dbf049fb2288b0: Neither blocks nor log segments found. Creating new log.
I20260926 11:33:50.214740 18991 log.cc:824] T 71b5c7a4316548ab96131db94a1cd0d1 P 1f63de4907154edfae24c3f8275496b0: Log is configured to *not* fsync() on all Append() calls
I20260926 11:33:50.214740 18990 log.cc:824] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17: Log is configured to *not* fsync() on all Append() calls
I20260926 11:33:50.215639 18992 log.cc:824] T 71b5c7a4316548ab96131db94a1cd0d1 P 31a855c7ddb44bacb6dbf049fb2288b0: Log is configured to *not* fsync() on all Append() calls
I20260926 11:33:50.216778 18991 tablet_bootstrap.cc:492] T 71b5c7a4316548ab96131db94a1cd0d1 P 1f63de4907154edfae24c3f8275496b0: No bootstrap required, opened a new log
I20260926 11:33:50.216990 18991 ts_tablet_manager.cc:1397] T 71b5c7a4316548ab96131db94a1cd0d1 P 1f63de4907154edfae24c3f8275496b0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 11:33:50.217199 18990 tablet_bootstrap.cc:492] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17: No bootstrap required, opened a new log
I20260926 11:33:50.217362 18990 ts_tablet_manager.cc:1397] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 11:33:50.217494 18992 tablet_bootstrap.cc:492] T 71b5c7a4316548ab96131db94a1cd0d1 P 31a855c7ddb44bacb6dbf049fb2288b0: No bootstrap required, opened a new log
I20260926 11:33:50.217662 18992 ts_tablet_manager.cc:1397] T 71b5c7a4316548ab96131db94a1cd0d1 P 31a855c7ddb44bacb6dbf049fb2288b0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 11:33:50.220422 18992 raft_consensus.cc:348] T 71b5c7a4316548ab96131db94a1cd0d1 P 31a855c7ddb44bacb6dbf049fb2288b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9ec79fcd264daa85b7cc62ceb9ac17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40747 } } peers { permanent_uuid: "1f63de4907154edfae24c3f8275496b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33581 } } peers { permanent_uuid: "31a855c7ddb44bacb6dbf049fb2288b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35265 } }
I20260926 11:33:50.220666 18992 raft_consensus.cc:374] T 71b5c7a4316548ab96131db94a1cd0d1 P 31a855c7ddb44bacb6dbf049fb2288b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:33:50.220745 18992 raft_consensus.cc:729] T 71b5c7a4316548ab96131db94a1cd0d1 P 31a855c7ddb44bacb6dbf049fb2288b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31a855c7ddb44bacb6dbf049fb2288b0, State: Initialized, Role: FOLLOWER
I20260926 11:33:50.221194 18992 consensus_queue.cc:260] T 71b5c7a4316548ab96131db94a1cd0d1 P 31a855c7ddb44bacb6dbf049fb2288b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9ec79fcd264daa85b7cc62ceb9ac17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40747 } } peers { permanent_uuid: "1f63de4907154edfae24c3f8275496b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33581 } } peers { permanent_uuid: "31a855c7ddb44bacb6dbf049fb2288b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35265 } }
I20260926 11:33:50.221369 18990 raft_consensus.cc:348] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9ec79fcd264daa85b7cc62ceb9ac17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40747 } } peers { permanent_uuid: "1f63de4907154edfae24c3f8275496b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33581 } } peers { permanent_uuid: "31a855c7ddb44bacb6dbf049fb2288b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35265 } }
I20260926 11:33:50.221686 18990 raft_consensus.cc:374] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:33:50.221752 18990 raft_consensus.cc:729] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f9ec79fcd264daa85b7cc62ceb9ac17, State: Initialized, Role: FOLLOWER
I20260926 11:33:50.221828 18973 heartbeater.cc:503] Master 127.0.98.62:36313 was elected leader, sending a full tablet report...
I20260926 11:33:50.222150 18992 ts_tablet_manager.cc:1428] T 71b5c7a4316548ab96131db94a1cd0d1 P 31a855c7ddb44bacb6dbf049fb2288b0: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20260926 11:33:50.222146 18990 consensus_queue.cc:260] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9ec79fcd264daa85b7cc62ceb9ac17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40747 } } peers { permanent_uuid: "1f63de4907154edfae24c3f8275496b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33581 } } peers { permanent_uuid: "31a855c7ddb44bacb6dbf049fb2288b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35265 } }
I20260926 11:33:50.223076 18990 ts_tablet_manager.cc:1428] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 11:33:50.223033 18991 raft_consensus.cc:348] T 71b5c7a4316548ab96131db94a1cd0d1 P 1f63de4907154edfae24c3f8275496b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9ec79fcd264daa85b7cc62ceb9ac17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40747 } } peers { permanent_uuid: "1f63de4907154edfae24c3f8275496b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33581 } } peers { permanent_uuid: "31a855c7ddb44bacb6dbf049fb2288b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35265 } }
I20260926 11:33:50.223342 18991 raft_consensus.cc:374] T 71b5c7a4316548ab96131db94a1cd0d1 P 1f63de4907154edfae24c3f8275496b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:33:50.223412 18839 heartbeater.cc:503] Master 127.0.98.62:36313 was elected leader, sending a full tablet report...
I20260926 11:33:50.223443 18991 raft_consensus.cc:729] T 71b5c7a4316548ab96131db94a1cd0d1 P 1f63de4907154edfae24c3f8275496b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f63de4907154edfae24c3f8275496b0, State: Initialized, Role: FOLLOWER
I20260926 11:33:50.223865 18991 consensus_queue.cc:260] T 71b5c7a4316548ab96131db94a1cd0d1 P 1f63de4907154edfae24c3f8275496b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9ec79fcd264daa85b7cc62ceb9ac17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40747 } } peers { permanent_uuid: "1f63de4907154edfae24c3f8275496b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33581 } } peers { permanent_uuid: "31a855c7ddb44bacb6dbf049fb2288b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35265 } }
I20260926 11:33:50.224468 18705 heartbeater.cc:503] Master 127.0.98.62:36313 was elected leader, sending a full tablet report...
I20260926 11:33:50.224763 18991 ts_tablet_manager.cc:1428] T 71b5c7a4316548ab96131db94a1cd0d1 P 1f63de4907154edfae24c3f8275496b0: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260926 11:33:50.295025 18997 raft_consensus.cc:482] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:33:50.295282 18997 raft_consensus.cc:504] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9ec79fcd264daa85b7cc62ceb9ac17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40747 } } peers { permanent_uuid: "1f63de4907154edfae24c3f8275496b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33581 } } peers { permanent_uuid: "31a855c7ddb44bacb6dbf049fb2288b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35265 } }
I20260926 11:33:50.296540 18997 leader_election.cc:288] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f63de4907154edfae24c3f8275496b0 (127.0.98.1:33581), 31a855c7ddb44bacb6dbf049fb2288b0 (127.0.98.3:35265)
I20260926 11:33:50.300176 18841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:50.296797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35265 (local address 127.0.98.2:34943)
0926 11:33:50.296924 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:50.296939 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:50.297038 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:50.297321 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:50.297324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:50.297342 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:50.297460 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:50.297466 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.298247 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.298252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:50.299044 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:50.299052 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.299716 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.299720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:50.299852 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:50.299870 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 11:33:50.299973 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 11:33:50.300045 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 11:33:50.300176 18989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:50.296941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34943 (local address 127.0.98.3:35265)
0926 11:33:50.297104 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:50.297107 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:50.297121 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:33:50.297168 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:50.297170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:50.297210 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:50.297307 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:50.297593 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.298135 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.299175 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.299570 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.299701 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:50.299785 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:50.299992 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 11:33:50.300060 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 11:33:50.300604 18999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:50.296804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33581 (local address 127.0.98.2:48617)
0926 11:33:50.297372 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:50.297383 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:50.297507 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:50.297786 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:50.297789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:50.297803 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:50.297951 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:50.297957 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.298680 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.298686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:50.299437 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:50.299445 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.300139 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.300142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:50.300242 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:50.300275 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 11:33:50.300394 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 11:33:50.300456 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":243,"thread_start_us":161,"threads_started":1}
I20260926 11:33:50.300670 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:50.296912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48617 (local address 127.0.98.1:33581)
0926 11:33:50.297032 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:50.297036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:50.297468 (+ 432us) server_negotiation.cc:407] Connection header received
0926 11:33:50.297631 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:50.297634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:50.297693 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:50.297776 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:50.298069 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.298561 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.299560 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.300024 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.300137 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:50.300194 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:50.300451 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 11:33:50.300528 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 11:33:50.300904 18927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b5c7a4316548ab96131db94a1cd0d1" candidate_uuid: "9f9ec79fcd264daa85b7cc62ceb9ac17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a855c7ddb44bacb6dbf049fb2288b0" is_pre_election: true
I20260926 11:33:50.301230 18927 raft_consensus.cc:2454] T 71b5c7a4316548ab96131db94a1cd0d1 P 31a855c7ddb44bacb6dbf049fb2288b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f9ec79fcd264daa85b7cc62ceb9ac17 in term 0.
I20260926 11:33:50.301270 18659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b5c7a4316548ab96131db94a1cd0d1" candidate_uuid: "9f9ec79fcd264daa85b7cc62ceb9ac17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f63de4907154edfae24c3f8275496b0" is_pre_election: true
I20260926 11:33:50.301529 18659 raft_consensus.cc:2454] T 71b5c7a4316548ab96131db94a1cd0d1 P 1f63de4907154edfae24c3f8275496b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f9ec79fcd264daa85b7cc62ceb9ac17 in term 0.
I20260926 11:33:50.301797 18727 leader_election.cc:302] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31a855c7ddb44bacb6dbf049fb2288b0, 9f9ec79fcd264daa85b7cc62ceb9ac17; no voters:
I20260926 11:33:50.302101 18997 raft_consensus.cc:2781] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:33:50.302199 18997 raft_consensus.cc:482] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:33:50.302277 18997 raft_consensus.cc:3037] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:33:50.303323 18997 raft_consensus.cc:504] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9ec79fcd264daa85b7cc62ceb9ac17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40747 } } peers { permanent_uuid: "1f63de4907154edfae24c3f8275496b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33581 } } peers { permanent_uuid: "31a855c7ddb44bacb6dbf049fb2288b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35265 } }
I20260926 11:33:50.303679 18997 leader_election.cc:288] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17 [CANDIDATE]: Term 1 election: Requested vote from peers 1f63de4907154edfae24c3f8275496b0 (127.0.98.1:33581), 31a855c7ddb44bacb6dbf049fb2288b0 (127.0.98.3:35265)
I20260926 11:33:50.304164 18927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b5c7a4316548ab96131db94a1cd0d1" candidate_uuid: "9f9ec79fcd264daa85b7cc62ceb9ac17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31a855c7ddb44bacb6dbf049fb2288b0"
I20260926 11:33:50.304168 18659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71b5c7a4316548ab96131db94a1cd0d1" candidate_uuid: "9f9ec79fcd264daa85b7cc62ceb9ac17" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f63de4907154edfae24c3f8275496b0"
I20260926 11:33:50.304348 18659 raft_consensus.cc:3037] T 71b5c7a4316548ab96131db94a1cd0d1 P 1f63de4907154edfae24c3f8275496b0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:33:50.304332 18927 raft_consensus.cc:3037] T 71b5c7a4316548ab96131db94a1cd0d1 P 31a855c7ddb44bacb6dbf049fb2288b0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:33:50.305734 18659 raft_consensus.cc:2454] T 71b5c7a4316548ab96131db94a1cd0d1 P 1f63de4907154edfae24c3f8275496b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f9ec79fcd264daa85b7cc62ceb9ac17 in term 1.
I20260926 11:33:50.305734 18927 raft_consensus.cc:2454] T 71b5c7a4316548ab96131db94a1cd0d1 P 31a855c7ddb44bacb6dbf049fb2288b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f9ec79fcd264daa85b7cc62ceb9ac17 in term 1.
I20260926 11:33:50.306092 18726 leader_election.cc:302] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f63de4907154edfae24c3f8275496b0, 9f9ec79fcd264daa85b7cc62ceb9ac17; no voters:
I20260926 11:33:50.306303 18997 raft_consensus.cc:2781] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:33:50.306540 18997 raft_consensus.cc:686] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17 [term 1 LEADER]: Becoming Leader. State: Replica: 9f9ec79fcd264daa85b7cc62ceb9ac17, State: Running, Role: LEADER
I20260926 11:33:50.306881 18997 consensus_queue.cc:237] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9ec79fcd264daa85b7cc62ceb9ac17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40747 } } peers { permanent_uuid: "1f63de4907154edfae24c3f8275496b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33581 } } peers { permanent_uuid: "31a855c7ddb44bacb6dbf049fb2288b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35265 } }
I20260926 11:33:50.310461 18514 catalog_manager.cc:5495] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f9ec79fcd264daa85b7cc62ceb9ac17 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "9f9ec79fcd264daa85b7cc62ceb9ac17" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f9ec79fcd264daa85b7cc62ceb9ac17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f63de4907154edfae24c3f8275496b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31a855c7ddb44bacb6dbf049fb2288b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35265 } health_report { overall_health: UNKNOWN } } }
I20260926 11:33:50.322747 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:50.319448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33586 (local address 127.0.98.2:40747)
0926 11:33:50.319574 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:50.319578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:50.319592 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:33:50.319676 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:50.319678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:50.319782 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:50.319892 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:33:50.320115 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.320626 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.321430 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.321643 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.321733 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:50.321977 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:33:50.322369 (+ 392us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:33:50.322493 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:50.322583 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 11:33:50.322616 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 11:33:50.328694 18659 raft_consensus.cc:1261] T 71b5c7a4316548ab96131db94a1cd0d1 P 1f63de4907154edfae24c3f8275496b0 [term 1 FOLLOWER]: Refusing update from remote peer 9f9ec79fcd264daa85b7cc62ceb9ac17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:33:50.328694 18927 raft_consensus.cc:1261] T 71b5c7a4316548ab96131db94a1cd0d1 P 31a855c7ddb44bacb6dbf049fb2288b0 [term 1 FOLLOWER]: Refusing update from remote peer 9f9ec79fcd264daa85b7cc62ceb9ac17: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:33:50.329391 19000 consensus_queue.cc:1035] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31a855c7ddb44bacb6dbf049fb2288b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:33:50.329638 18997 consensus_queue.cc:1035] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f63de4907154edfae24c3f8275496b0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:33:50.342968 19004 mvcc.cc:205] Tried to move back new op lower bound from 7333570274618888192 to 7333570274539868160. Current Snapshot: MvccSnapshot[applied={T|T < 7333570274618888192 or (T in {7333570274618888192})}]
I20260926 11:33:50.348168 18514 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:33:50.348322 18514 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50336:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:33:50.350695 18514 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00296ffe499e41b096bc5fafbbf85f16: Sending DeleteTablet for 3 replicas of tablet 71b5c7a4316548ab96131db94a1cd0d1
I20260926 11:33:50.351524 18773 tablet_service.cc:1514] Processing DeleteTablet for tablet 71b5c7a4316548ab96131db94a1cd0d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:33:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33580
I20260926 11:33:50.351524 18639 tablet_service.cc:1514] Processing DeleteTablet for tablet 71b5c7a4316548ab96131db94a1cd0d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:33:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41716
I20260926 11:33:50.352094 18907 tablet_service.cc:1514] Processing DeleteTablet for tablet 71b5c7a4316548ab96131db94a1cd0d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:33:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39770
I20260926 11:33:50.352584 19014 tablet_replica.cc:330] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17: stopping tablet replica
I20260926 11:33:50.352612 19015 tablet_replica.cc:330] T 71b5c7a4316548ab96131db94a1cd0d1 P 31a855c7ddb44bacb6dbf049fb2288b0: stopping tablet replica
I20260926 11:33:50.352689 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 18576
I20260926 11:33:50.352919 19014 raft_consensus.cc:2229] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:33:50.353330 19014 raft_consensus.cc:2258] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:33:50.355705 19015 raft_consensus.cc:2229] T 71b5c7a4316548ab96131db94a1cd0d1 P 31a855c7ddb44bacb6dbf049fb2288b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:33:50.355813 19014 ts_tablet_manager.cc:1905] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:33:50.356163 19015 raft_consensus.cc:2258] T 71b5c7a4316548ab96131db94a1cd0d1 P 31a855c7ddb44bacb6dbf049fb2288b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:33:50.357752 19015 ts_tablet_manager.cc:1905] T 71b5c7a4316548ab96131db94a1cd0d1 P 31a855c7ddb44bacb6dbf049fb2288b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:33:50.358537 19014 ts_tablet_manager.cc:1918] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:33:50.358676 19014 log.cc:1196] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-1/wal/wals/71b5c7a4316548ab96131db94a1cd0d1
I20260926 11:33:50.359066 19014 ts_tablet_manager.cc:1939] T 71b5c7a4316548ab96131db94a1cd0d1 P 9f9ec79fcd264daa85b7cc62ceb9ac17: Deleting consensus metadata
I20260926 11:33:50.360314 18501 catalog_manager.cc:4841] TS 9f9ec79fcd264daa85b7cc62ceb9ac17 (127.0.98.2:40747): tablet 71b5c7a4316548ab96131db94a1cd0d1 (table test-table [id=696260f39ed04b268a66efc9a9573ea7]) successfully deleted
W20260926 11:33:50.360821 18498 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33581 (error 108)
I20260926 11:33:50.360949 19015 ts_tablet_manager.cc:1918] T 71b5c7a4316548ab96131db94a1cd0d1 P 31a855c7ddb44bacb6dbf049fb2288b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:33:50.361047 19015 log.cc:1196] T 71b5c7a4316548ab96131db94a1cd0d1 P 31a855c7ddb44bacb6dbf049fb2288b0: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790422180210734-392-0/minicluster-data/ts-2/wal/wals/71b5c7a4316548ab96131db94a1cd0d1
I20260926 11:33:50.361048 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 18710
I20260926 11:33:50.361361 19015 ts_tablet_manager.cc:1939] T 71b5c7a4316548ab96131db94a1cd0d1 P 31a855c7ddb44bacb6dbf049fb2288b0: Deleting consensus metadata
I20260926 11:33:50.363107 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:50.362791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33581 (local address 127.0.0.1:41732)
0926 11:33:50.362951 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:50.363022 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 11:33:50.363274 18498 catalog_manager.cc:4584] TS 1f63de4907154edfae24c3f8275496b0 (127.0.98.1:33581): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 71b5c7a4316548ab96131db94a1cd0d1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33581: connect: Connection refused (error 111)
I20260926 11:33:50.363852 18499 catalog_manager.cc:4841] TS 31a855c7ddb44bacb6dbf049fb2288b0 (127.0.98.3:35265): tablet 71b5c7a4316548ab96131db94a1cd0d1 (table test-table [id=696260f39ed04b268a66efc9a9573ea7]) successfully deleted
I20260926 11:33:50.368171 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 18844
I20260926 11:33:50.374061 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 18482
2026-09-26T11:33:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2337 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T11:33:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:33:50Z Disabled control of system clock
I20260926 11:33:50.404513 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43025
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 11:33:50.512769 19020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:50.513041 19020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:50.513077 19020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:50.517025 19020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:33:50.517081 19020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:50.517096 19020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:50.517113 19020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:33:50.517128 19020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:33:50.521785 19020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43025
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:50.522807 19020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:50.524158 19020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:33:50.532353 19025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:50.532374 19020 server_base.cc:1034] running on GCE node
W20260926 11:33:50.532418 19028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:50.532383 19026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:50.533210 19020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:50.534272 19020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:50.535429 19020 hybrid_clock.cc:648] HybridClock initialized: now 1790422430535411 us; error 33 us; skew 500 ppm
I20260926 11:33:50.537228 19020 webserver.cc:460] Webserver started at http://127.0.98.62:35879/ using document root <none> and password file <none>
I20260926 11:33:50.537803 19020 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:50.537865 19020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:50.538085 19020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:33:50.540164 19020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "acaa5b6abfe44f7290fe175273594c63"
format_stamp: "Formatted at 2026-09-26 11:33:50 on dist-test-slave-mrck"
server_key: "a7c182584385d7699326008b2f9a6096"
server_key_iv: "293864bb6f939918dbfe11fffdbca078"
server_key_version: "encryptionkey@0"
I20260926 11:33:50.540717 19020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "acaa5b6abfe44f7290fe175273594c63"
format_stamp: "Formatted at 2026-09-26 11:33:50 on dist-test-slave-mrck"
server_key: "a7c182584385d7699326008b2f9a6096"
server_key_iv: "293864bb6f939918dbfe11fffdbca078"
server_key_version: "encryptionkey@0"
I20260926 11:33:50.544309 19020 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 11:33:50.546661 19034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:50.548026 19020 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:33:50.548146 19020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "acaa5b6abfe44f7290fe175273594c63"
format_stamp: "Formatted at 2026-09-26 11:33:50 on dist-test-slave-mrck"
server_key: "a7c182584385d7699326008b2f9a6096"
server_key_iv: "293864bb6f939918dbfe11fffdbca078"
server_key_version: "encryptionkey@0"
I20260926 11:33:50.548249 19020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:50.564946 19020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:50.565703 19020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:50.565908 19020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:50.572886 19020 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43025
I20260926 11:33:50.572911 19086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43025 every 8 connection(s)
I20260926 11:33:50.573889 19020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:33:50.576970 19087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:33:50.580288 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 19020
I20260926 11:33:50.580428 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:33:50.580745 392 external_mini_cluster.cc:1428] Setting key 8deba87269affd43b90c2aa105b04abc
I20260926 11:33:50.583029 19087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acaa5b6abfe44f7290fe175273594c63: Bootstrap starting.
I20260926 11:33:50.585280 19087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P acaa5b6abfe44f7290fe175273594c63: Neither blocks nor log segments found. Creating new log.
I20260926 11:33:50.586024 19087 log.cc:824] T 00000000000000000000000000000000 P acaa5b6abfe44f7290fe175273594c63: Log is configured to *not* fsync() on all Append() calls
I20260926 11:33:50.587996 19087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P acaa5b6abfe44f7290fe175273594c63: No bootstrap required, opened a new log
I20260926 11:33:50.590210 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:50.582159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35506 (local address 127.0.98.62:43025)
0926 11:33:50.582797 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:50.582812 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:50.582847 (+ 35us) server_negotiation.cc:407] Connection header received
0926 11:33:50.583553 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:50.583571 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:50.584045 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:50.584380 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:50.584818 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.585851 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.586829 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.587311 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.587483 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:50.588441 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:50.588446 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:50.588461 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:33:50.588847 (+ 386us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:50.588901 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:33:50.588920 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:50.589022 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:50.589183 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 11:33:50.589476 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":120,"threads_started":1}
I20260926 11:33:50.591991 19087 raft_consensus.cc:348] T 00000000000000000000000000000000 P acaa5b6abfe44f7290fe175273594c63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acaa5b6abfe44f7290fe175273594c63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43025 } }
I20260926 11:33:50.592264 19087 raft_consensus.cc:374] T 00000000000000000000000000000000 P acaa5b6abfe44f7290fe175273594c63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:33:50.592329 19087 raft_consensus.cc:729] T 00000000000000000000000000000000 P acaa5b6abfe44f7290fe175273594c63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acaa5b6abfe44f7290fe175273594c63, State: Initialized, Role: FOLLOWER
I20260926 11:33:50.592789 19087 consensus_queue.cc:260] T 00000000000000000000000000000000 P acaa5b6abfe44f7290fe175273594c63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acaa5b6abfe44f7290fe175273594c63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43025 } }
I20260926 11:33:50.592937 19087 raft_consensus.cc:388] T 00000000000000000000000000000000 P acaa5b6abfe44f7290fe175273594c63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:33:50.593020 19087 raft_consensus.cc:482] T 00000000000000000000000000000000 P acaa5b6abfe44f7290fe175273594c63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:33:50.593135 19087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P acaa5b6abfe44f7290fe175273594c63 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:33:50.594063 19087 raft_consensus.cc:504] T 00000000000000000000000000000000 P acaa5b6abfe44f7290fe175273594c63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acaa5b6abfe44f7290fe175273594c63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43025 } }
I20260926 11:33:50.594441 19087 leader_election.cc:302] T 00000000000000000000000000000000 P acaa5b6abfe44f7290fe175273594c63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: acaa5b6abfe44f7290fe175273594c63; no voters:
I20260926 11:33:50.594755 19087 leader_election.cc:288] T 00000000000000000000000000000000 P acaa5b6abfe44f7290fe175273594c63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:33:50.594871 19092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P acaa5b6abfe44f7290fe175273594c63 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:33:50.595104 19092 raft_consensus.cc:686] T 00000000000000000000000000000000 P acaa5b6abfe44f7290fe175273594c63 [term 1 LEADER]: Becoming Leader. State: Replica: acaa5b6abfe44f7290fe175273594c63, State: Running, Role: LEADER
I20260926 11:33:50.595465 19092 consensus_queue.cc:237] T 00000000000000000000000000000000 P acaa5b6abfe44f7290fe175273594c63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acaa5b6abfe44f7290fe175273594c63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43025 } }
I20260926 11:33:50.596043 19087 sys_catalog.cc:564] T 00000000000000000000000000000000 P acaa5b6abfe44f7290fe175273594c63 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:33:50.597254 19093 sys_catalog.cc:455] T 00000000000000000000000000000000 P acaa5b6abfe44f7290fe175273594c63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "acaa5b6abfe44f7290fe175273594c63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acaa5b6abfe44f7290fe175273594c63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43025 } } }
I20260926 11:33:50.597239 19094 sys_catalog.cc:455] T 00000000000000000000000000000000 P acaa5b6abfe44f7290fe175273594c63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader acaa5b6abfe44f7290fe175273594c63. Latest consensus state: current_term: 1 leader_uuid: "acaa5b6abfe44f7290fe175273594c63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "acaa5b6abfe44f7290fe175273594c63" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43025 } } }
I20260926 11:33:50.597393 19094 sys_catalog.cc:458] T 00000000000000000000000000000000 P acaa5b6abfe44f7290fe175273594c63 [sys.catalog]: This master's current role is: LEADER
I20260926 11:33:50.597393 19093 sys_catalog.cc:458] T 00000000000000000000000000000000 P acaa5b6abfe44f7290fe175273594c63 [sys.catalog]: This master's current role is: LEADER
I20260926 11:33:50.598318 19101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:33:50.601074 19101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:33:50.606709 19101 catalog_manager.cc:1347] Generated new cluster ID: a70ea7b269c84a0fb1a8427cb59ebae3
I20260926 11:33:50.606799 19101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:33:50.625006 19101 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:33:50.626317 19101 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:33:50.631321 19101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P acaa5b6abfe44f7290fe175273594c63: Generated new TSK 0
I20260926 11:33:50.632066 19101 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:33:50.639087 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43025
--builtin_ntp_servers=127.0.98.20:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 11:33:50.754877 19111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:50.755213 19111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:50.755302 19111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:50.759511 19111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:50.759611 19111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:50.759708 19111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:33:50.764833 19111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:50.765960 19111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:50.767357 19111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:33:50.777254 19116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:50.777345 19111 server_base.cc:1034] running on GCE node
W20260926 11:33:50.777277 19117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:50.777284 19119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:50.778023 19111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:50.778733 19111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:50.779955 19111 hybrid_clock.cc:648] HybridClock initialized: now 1790422430779894 us; error 77 us; skew 500 ppm
I20260926 11:33:50.787349 19111 webserver.cc:460] Webserver started at http://127.0.98.1:42851/ using document root <none> and password file <none>
I20260926 11:33:50.788051 19111 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:50.788121 19111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:50.788375 19111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:33:50.790534 19111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-0/data/instance:
uuid: "00688963452e4d8581a054fb2b5f018f"
format_stamp: "Formatted at 2026-09-26 11:33:50 on dist-test-slave-mrck"
server_key: "85a5c8ffe8e261b126ffb69fc96e42e9"
server_key_iv: "3264ac82499d43ee01c3ad4cb37e862a"
server_key_version: "encryptionkey@0"
I20260926 11:33:50.791109 19111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "00688963452e4d8581a054fb2b5f018f"
format_stamp: "Formatted at 2026-09-26 11:33:50 on dist-test-slave-mrck"
server_key: "85a5c8ffe8e261b126ffb69fc96e42e9"
server_key_iv: "3264ac82499d43ee01c3ad4cb37e862a"
server_key_version: "encryptionkey@0"
I20260926 11:33:50.794930 19111 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 11:33:50.797900 19125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:50.799263 19111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 11:33:50.799391 19111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "00688963452e4d8581a054fb2b5f018f"
format_stamp: "Formatted at 2026-09-26 11:33:50 on dist-test-slave-mrck"
server_key: "85a5c8ffe8e261b126ffb69fc96e42e9"
server_key_iv: "3264ac82499d43ee01c3ad4cb37e862a"
server_key_version: "encryptionkey@0"
I20260926 11:33:50.799511 19111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:50.820190 19111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:50.820932 19111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:50.821142 19111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:50.821767 19111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:33:50.822724 19111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:33:50.822770 19111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:50.822837 19111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:33:50.822868 19111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:50.832031 19111 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46827
I20260926 11:33:50.832054 19238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46827 every 8 connection(s)
I20260926 11:33:50.832968 19111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:33:50.836055 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 19111
I20260926 11:33:50.836205 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:33:50.836503 392 external_mini_cluster.cc:1428] Setting key af8fe2d5c2c84b9b0cd59cb5e34468c3
I20260926 11:33:50.838634 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43025
--builtin_ntp_servers=127.0.98.20:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 11:33:50.842620 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:50.834792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45899 (local address 127.0.98.62:43025)
0926 11:33:50.834930 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:50.834934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:50.836088 (+ 1154us) server_negotiation.cc:407] Connection header received
0926 11:33:50.837785 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:50.837792 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:50.837852 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:50.837937 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:50.838934 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.839465 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.840684 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.840923 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.841044 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:50.842134 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:50.842138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:50.842141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:33:50.842153 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:50.842180 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:33:50.842189 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:50.842268 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:50.842421 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 11:33:50.842481 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 11:33:50.843480 19241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:50.835199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43025 (local address 127.0.98.1:45899)
0926 11:33:50.835931 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:50.835973 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:50.837531 (+ 1558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:50.838158 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:50.838165 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:50.838254 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:33:50.838774 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:50.838794 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.839597 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.839601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:50.840558 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:50.840568 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:50.841082 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:50.841085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:50.841250 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:50.841865 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:33:50.841867 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:33:50.841908 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:33:50.841972 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:33:50.842270 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:33:50.842282 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:33:50.842289 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 11:33:50.842421 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 11:33:50.842720 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":104,"threads_started":1}
I20260926 11:33:50.844755 19239 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43025
I20260926 11:33:50.845022 19239 heartbeater.cc:464] Registering TS with master...
I20260926 11:33:50.845696 19239 heartbeater.cc:511] Master 127.0.98.62:43025 requested a full tablet report, sending...
I20260926 11:33:50.847497 19051 ts_manager.cc:194] Registered new tserver with Master: 00688963452e4d8581a054fb2b5f018f (127.0.98.1:46827)
I20260926 11:33:50.849237 19051 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:45899
W20260926 11:33:50.968863 19242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:50.969132 19242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:50.969172 19242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:50.973086 19242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:50.973135 19242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:50.973227 19242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:33:50.977880 19242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:50.978924 19242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:50.980331 19242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:33:50.989692 19247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:50.989705 19250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:50.989703 19248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:50.989886 19242 server_base.cc:1034] running on GCE node
I20260926 11:33:50.990491 19242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:50.991111 19242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:50.992265 19242 hybrid_clock.cc:648] HybridClock initialized: now 1790422430992231 us; error 44 us; skew 500 ppm
I20260926 11:33:51.000559 19242 webserver.cc:460] Webserver started at http://127.0.98.2:38491/ using document root <none> and password file <none>
I20260926 11:33:51.001178 19242 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:51.001241 19242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:51.001492 19242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:33:51.003795 19242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-1/data/instance:
uuid: "ca6e0c033f25422b820518ab00262448"
format_stamp: "Formatted at 2026-09-26 11:33:50 on dist-test-slave-mrck"
server_key: "ced05e0dcb8eb71d3beb38e195ed51a9"
server_key_iv: "a307f25ae3005428221f5e4795376dbb"
server_key_version: "encryptionkey@0"
I20260926 11:33:51.004398 19242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ca6e0c033f25422b820518ab00262448"
format_stamp: "Formatted at 2026-09-26 11:33:50 on dist-test-slave-mrck"
server_key: "ced05e0dcb8eb71d3beb38e195ed51a9"
server_key_iv: "a307f25ae3005428221f5e4795376dbb"
server_key_version: "encryptionkey@0"
I20260926 11:33:51.008185 19242 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:33:51.010627 19256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:51.011845 19242 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:33:51.011982 19242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "ca6e0c033f25422b820518ab00262448"
format_stamp: "Formatted at 2026-09-26 11:33:50 on dist-test-slave-mrck"
server_key: "ced05e0dcb8eb71d3beb38e195ed51a9"
server_key_iv: "a307f25ae3005428221f5e4795376dbb"
server_key_version: "encryptionkey@0"
I20260926 11:33:51.012091 19242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:51.022189 19242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:51.022816 19242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:51.023006 19242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:51.023555 19242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:33:51.024518 19242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:33:51.024587 19242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:51.024654 19242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:33:51.024701 19242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:51.033725 19242 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46589
I20260926 11:33:51.033746 19369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46589 every 8 connection(s)
I20260926 11:33:51.034677 19242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:33:51.035079 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 19242
I20260926 11:33:51.035225 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:33:51.035588 392 external_mini_cluster.cc:1428] Setting key e4fa7427e1a49d3711c112cbbfc77b83
I20260926 11:33:51.037922 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43025
--builtin_ntp_servers=127.0.98.20:46477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 11:33:51.046095 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:51.036803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41705 (local address 127.0.98.62:43025)
0926 11:33:51.036959 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:51.036964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:51.038301 (+ 1337us) server_negotiation.cc:407] Connection header received
0926 11:33:51.040105 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:51.040108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:51.040155 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:51.040222 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:51.041444 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:51.041961 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:51.043934 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:51.044167 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:51.044258 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:51.045632 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:51.045635 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:51.045637 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:33:51.045649 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:51.045669 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:33:51.045677 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:51.045732 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:51.045928 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 11:33:51.045982 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 11:33:51.047472 19372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:51.037371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43025 (local address 127.0.98.2:41705)
0926 11:33:51.038155 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:51.038213 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:51.039797 (+ 1584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:51.040437 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:51.040449 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:51.040565 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:33:51.041290 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:51.041311 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:51.042088 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:51.042095 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:51.043779 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:51.043796 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:51.044338 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:51.044346 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:51.044594 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:51.045284 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:33:51.045288 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:33:51.045351 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:33:51.045450 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:33:51.045761 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:33:51.045774 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:33:51.045781 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 11:33:51.045927 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 11:33:51.046279 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":137,"threads_started":1}
I20260926 11:33:51.048775 19370 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43025
I20260926 11:33:51.049031 19370 heartbeater.cc:464] Registering TS with master...
I20260926 11:33:51.049810 19370 heartbeater.cc:511] Master 127.0.98.62:43025 requested a full tablet report, sending...
I20260926 11:33:51.050891 19051 ts_manager.cc:194] Registered new tserver with Master: ca6e0c033f25422b820518ab00262448 (127.0.98.2:46589)
I20260926 11:33:51.051469 19051 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:41705
W20260926 11:33:51.152460 19373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:51.152717 19373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:51.152774 19373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:51.156944 19373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:51.156999 19373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:51.157080 19373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:33:51.161825 19373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:51.162873 19373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:51.164333 19373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:33:51.173646 19379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:51.173758 19373 server_base.cc:1034] running on GCE node
W20260926 11:33:51.173640 19381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:51.174010 19378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:51.174417 19373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:51.175037 19373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:51.176188 19373 hybrid_clock.cc:648] HybridClock initialized: now 1790422431176170 us; error 29 us; skew 500 ppm
I20260926 11:33:51.177923 19373 webserver.cc:460] Webserver started at http://127.0.98.3:33139/ using document root <none> and password file <none>
I20260926 11:33:51.178484 19373 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:51.178543 19373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:51.178773 19373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:33:51.180817 19373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-2/data/instance:
uuid: "3b4902991ebe46ae8864959c708e0fe1"
format_stamp: "Formatted at 2026-09-26 11:33:51 on dist-test-slave-mrck"
server_key: "8865c3d5720e6cc75616e2dc75aa3190"
server_key_iv: "3cb0b3686f093b1ca3f49b9fa98a5e6a"
server_key_version: "encryptionkey@0"
I20260926 11:33:51.181356 19373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3b4902991ebe46ae8864959c708e0fe1"
format_stamp: "Formatted at 2026-09-26 11:33:51 on dist-test-slave-mrck"
server_key: "8865c3d5720e6cc75616e2dc75aa3190"
server_key_iv: "3cb0b3686f093b1ca3f49b9fa98a5e6a"
server_key_version: "encryptionkey@0"
I20260926 11:33:51.184875 19373 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:33:51.187387 19387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:51.188614 19373 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:33:51.188740 19373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "3b4902991ebe46ae8864959c708e0fe1"
format_stamp: "Formatted at 2026-09-26 11:33:51 on dist-test-slave-mrck"
server_key: "8865c3d5720e6cc75616e2dc75aa3190"
server_key_iv: "3cb0b3686f093b1ca3f49b9fa98a5e6a"
server_key_version: "encryptionkey@0"
I20260926 11:33:51.188846 19373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:51.227488 19373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:51.228243 19373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:51.228443 19373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:51.229069 19373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:33:51.230022 19373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:33:51.230068 19373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:51.230136 19373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:33:51.230175 19373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:51.238972 19373 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39831
I20260926 11:33:51.239014 19500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39831 every 8 connection(s)
I20260926 11:33:51.239941 19373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:33:51.245157 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 19373
I20260926 11:33:51.245277 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:33:51.245579 392 external_mini_cluster.cc:1428] Setting key a24fe9ff582446ed7c3cc8f65f801bba
I20260926 11:33:51.249660 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:51.241824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52677 (local address 127.0.98.62:43025)
0926 11:33:51.241966 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:51.241970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:51.243015 (+ 1045us) server_negotiation.cc:407] Connection header received
0926 11:33:51.244432 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:51.244434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:51.244482 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:51.244558 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:51.245572 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:51.246208 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:51.247496 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:51.247920 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:51.248053 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:51.249260 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:51.249262 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:51.249264 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:33:51.249275 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:51.249295 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:33:51.249302 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:51.249362 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:51.249501 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 11:33:51.249551 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 11:33:51.250622 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:51.242236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43025 (local address 127.0.98.3:52677)
0926 11:33:51.242868 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:51.242910 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:51.244243 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:51.244715 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:51.244722 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:51.244806 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:33:51.245438 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:51.245453 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:51.246336 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:51.246340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:51.247357 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:51.247366 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:51.248103 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:51.248109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:51.248332 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:51.249023 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:33:51.249025 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:33:51.249066 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:33:51.249130 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:33:51.249377 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:33:51.249383 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:33:51.249387 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 11:33:51.249483 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 11:33:51.249724 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":105,"threads_started":1}
I20260926 11:33:51.251850 19501 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43025
I20260926 11:33:51.252111 19501 heartbeater.cc:464] Registering TS with master...
I20260926 11:33:51.252653 19501 heartbeater.cc:511] Master 127.0.98.62:43025 requested a full tablet report, sending...
I20260926 11:33:51.253701 19051 ts_manager.cc:194] Registered new tserver with Master: 3b4902991ebe46ae8864959c708e0fe1 (127.0.98.3:39831)
I20260926 11:33:51.254274 19051 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:52677
I20260926 11:33:51.259430 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:33:51.264789 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:51.261333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35514 (local address 127.0.98.62:43025)
0926 11:33:51.261454 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:51.261457 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:51.261654 (+ 197us) server_negotiation.cc:407] Connection header received
0926 11:33:51.261747 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:51.261749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:51.261785 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:51.261854 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:51.262105 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:51.262578 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:51.263334 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:51.263540 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:51.263613 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:51.264401 (+ 788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:51.264405 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:51.264408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:33:51.264419 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:51.264438 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:33:51.264445 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:51.264543 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:51.264656 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 11:33:51.264688 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 11:33:51.267442 19051 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:33:51.270597 19051 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:33:51.285187 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:51.280131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39980 (local address 127.0.98.3:39831)
0926 11:33:51.280611 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:51.280623 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:51.280646 (+ 23us) server_negotiation.cc:407] Connection header received
0926 11:33:51.280728 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:51.280731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:51.280913 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:51.281346 (+ 433us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:51.281482 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:51.282460 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:51.283939 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:51.284462 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:51.284585 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:51.284786 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:51.284923 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 11:33:51.285021 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":161,"threads_started":1}
I20260926 11:33:51.285183 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:51.279071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46827 (local address 127.0.0.1:55126)
0926 11:33:51.279802 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:51.279814 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:51.279892 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:51.281035 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:51.281038 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:51.281055 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:51.281205 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:51.281211 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:51.282154 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:51.282161 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:51.283533 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:51.283542 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:51.284668 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:51.284672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:51.284789 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:51.284811 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 11:33:51.284875 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 11:33:51.285034 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":123,"thread_start_us":94,"threads_started":1}
I20260926 11:33:51.285495 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:51.279098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57828 (local address 127.0.98.2:46589)
0926 11:33:51.279552 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:51.279567 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:51.279952 (+ 385us) server_negotiation.cc:407] Connection header received
0926 11:33:51.280032 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:51.280038 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:51.280254 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:51.280425 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:51.280880 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:51.282090 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:51.283791 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:51.284529 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:51.284646 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:51.284821 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:51.285245 (+ 424us) server_negotiation.cc:299] Negotiation successful
0926 11:33:51.285331 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":129,"threads_started":1}
I20260926 11:33:51.285491 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:51.279698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55126 (local address 127.0.98.1:46827)
0926 11:33:51.280595 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:51.280607 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:51.280627 (+ 20us) server_negotiation.cc:407] Connection header received
0926 11:33:51.280693 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:51.280698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:51.280888 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:51.281056 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:51.281320 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:51.282039 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:51.283655 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:51.284529 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:51.285011 (+ 482us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:51.285190 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:51.285240 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 11:33:51.285330 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":794,"thread_start_us":103,"threads_started":1}
I20260926 11:33:51.285812 19512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:51.278991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46589 (local address 127.0.0.1:57828)
0926 11:33:51.279595 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:51.279642 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:51.279800 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:51.280601 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:51.280609 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:51.280632 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:51.280760 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:51.280766 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:51.282200 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:51.282203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:51.283656 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:51.283666 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:51.284880 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:51.284884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:51.284975 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:51.284995 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 11:33:51.285055 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 11:33:51.285122 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"mutex_wait_us":119,"thread_start_us":72,"threads_started":1}
I20260926 11:33:51.285826 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:51.279680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39831 (local address 127.0.0.1:39980)
0926 11:33:51.280237 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:51.280249 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:51.280350 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:51.281074 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:51.281077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:51.281096 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:51.281218 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:51.281224 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:51.282564 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:51.282570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:51.283821 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:51.283835 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:51.284585 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:51.284589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:51.284744 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:51.284786 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 11:33:51.284895 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 11:33:51.285653 (+ 758us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":111,"thread_start_us":115,"threads_started":1}
I20260926 11:33:51.288822 19304 tablet_service.cc:1467] Processing CreateTablet for tablet 76811f681a8746d9b89e8ee34ea98ffe (DEFAULT_TABLE table=test-table [id=1f4d8676d63f43ff9867b3b87c3a412d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:33:51.288846 19173 tablet_service.cc:1467] Processing CreateTablet for tablet 76811f681a8746d9b89e8ee34ea98ffe (DEFAULT_TABLE table=test-table [id=1f4d8676d63f43ff9867b3b87c3a412d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:33:51.289029 19435 tablet_service.cc:1467] Processing CreateTablet for tablet 76811f681a8746d9b89e8ee34ea98ffe (DEFAULT_TABLE table=test-table [id=1f4d8676d63f43ff9867b3b87c3a412d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:33:51.289820 19304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76811f681a8746d9b89e8ee34ea98ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:33:51.289878 19435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76811f681a8746d9b89e8ee34ea98ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:33:51.289894 19173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76811f681a8746d9b89e8ee34ea98ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:33:51.297793 19520 tablet_bootstrap.cc:492] T 76811f681a8746d9b89e8ee34ea98ffe P 3b4902991ebe46ae8864959c708e0fe1: Bootstrap starting.
I20260926 11:33:51.298044 19518 tablet_bootstrap.cc:492] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448: Bootstrap starting.
I20260926 11:33:51.298259 19519 tablet_bootstrap.cc:492] T 76811f681a8746d9b89e8ee34ea98ffe P 00688963452e4d8581a054fb2b5f018f: Bootstrap starting.
I20260926 11:33:51.300004 19518 tablet_bootstrap.cc:654] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448: Neither blocks nor log segments found. Creating new log.
I20260926 11:33:51.300173 19519 tablet_bootstrap.cc:654] T 76811f681a8746d9b89e8ee34ea98ffe P 00688963452e4d8581a054fb2b5f018f: Neither blocks nor log segments found. Creating new log.
I20260926 11:33:51.300172 19520 tablet_bootstrap.cc:654] T 76811f681a8746d9b89e8ee34ea98ffe P 3b4902991ebe46ae8864959c708e0fe1: Neither blocks nor log segments found. Creating new log.
I20260926 11:33:51.300787 19518 log.cc:824] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448: Log is configured to *not* fsync() on all Append() calls
I20260926 11:33:51.301025 19519 log.cc:824] T 76811f681a8746d9b89e8ee34ea98ffe P 00688963452e4d8581a054fb2b5f018f: Log is configured to *not* fsync() on all Append() calls
I20260926 11:33:51.301025 19520 log.cc:824] T 76811f681a8746d9b89e8ee34ea98ffe P 3b4902991ebe46ae8864959c708e0fe1: Log is configured to *not* fsync() on all Append() calls
I20260926 11:33:51.302940 19518 tablet_bootstrap.cc:492] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448: No bootstrap required, opened a new log
I20260926 11:33:51.302963 19519 tablet_bootstrap.cc:492] T 76811f681a8746d9b89e8ee34ea98ffe P 00688963452e4d8581a054fb2b5f018f: No bootstrap required, opened a new log
I20260926 11:33:51.302953 19520 tablet_bootstrap.cc:492] T 76811f681a8746d9b89e8ee34ea98ffe P 3b4902991ebe46ae8864959c708e0fe1: No bootstrap required, opened a new log
I20260926 11:33:51.303135 19519 ts_tablet_manager.cc:1397] T 76811f681a8746d9b89e8ee34ea98ffe P 00688963452e4d8581a054fb2b5f018f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 11:33:51.303135 19520 ts_tablet_manager.cc:1397] T 76811f681a8746d9b89e8ee34ea98ffe P 3b4902991ebe46ae8864959c708e0fe1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 11:33:51.303140 19518 ts_tablet_manager.cc:1397] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 11:33:51.306636 19518 raft_consensus.cc:348] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6e0c033f25422b820518ab00262448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46589 } } peers { permanent_uuid: "00688963452e4d8581a054fb2b5f018f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46827 } } peers { permanent_uuid: "3b4902991ebe46ae8864959c708e0fe1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 } }
I20260926 11:33:51.306880 19518 raft_consensus.cc:374] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:33:51.306917 19518 raft_consensus.cc:729] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca6e0c033f25422b820518ab00262448, State: Initialized, Role: FOLLOWER
I20260926 11:33:51.307402 19518 consensus_queue.cc:260] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6e0c033f25422b820518ab00262448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46589 } } peers { permanent_uuid: "00688963452e4d8581a054fb2b5f018f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46827 } } peers { permanent_uuid: "3b4902991ebe46ae8864959c708e0fe1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 } }
I20260926 11:33:51.307524 19519 raft_consensus.cc:348] T 76811f681a8746d9b89e8ee34ea98ffe P 00688963452e4d8581a054fb2b5f018f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6e0c033f25422b820518ab00262448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46589 } } peers { permanent_uuid: "00688963452e4d8581a054fb2b5f018f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46827 } } peers { permanent_uuid: "3b4902991ebe46ae8864959c708e0fe1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 } }
I20260926 11:33:51.307636 19520 raft_consensus.cc:348] T 76811f681a8746d9b89e8ee34ea98ffe P 3b4902991ebe46ae8864959c708e0fe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6e0c033f25422b820518ab00262448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46589 } } peers { permanent_uuid: "00688963452e4d8581a054fb2b5f018f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46827 } } peers { permanent_uuid: "3b4902991ebe46ae8864959c708e0fe1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 } }
I20260926 11:33:51.307873 19519 raft_consensus.cc:374] T 76811f681a8746d9b89e8ee34ea98ffe P 00688963452e4d8581a054fb2b5f018f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:33:51.307932 19520 raft_consensus.cc:374] T 76811f681a8746d9b89e8ee34ea98ffe P 3b4902991ebe46ae8864959c708e0fe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:33:51.307979 19519 raft_consensus.cc:729] T 76811f681a8746d9b89e8ee34ea98ffe P 00688963452e4d8581a054fb2b5f018f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00688963452e4d8581a054fb2b5f018f, State: Initialized, Role: FOLLOWER
I20260926 11:33:51.308173 19370 heartbeater.cc:503] Master 127.0.98.62:43025 was elected leader, sending a full tablet report...
I20260926 11:33:51.308418 19519 consensus_queue.cc:260] T 76811f681a8746d9b89e8ee34ea98ffe P 00688963452e4d8581a054fb2b5f018f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6e0c033f25422b820518ab00262448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46589 } } peers { permanent_uuid: "00688963452e4d8581a054fb2b5f018f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46827 } } peers { permanent_uuid: "3b4902991ebe46ae8864959c708e0fe1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 } }
I20260926 11:33:51.308565 19518 ts_tablet_manager.cc:1428] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260926 11:33:51.308678 19520 raft_consensus.cc:729] T 76811f681a8746d9b89e8ee34ea98ffe P 3b4902991ebe46ae8864959c708e0fe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b4902991ebe46ae8864959c708e0fe1, State: Initialized, Role: FOLLOWER
I20260926 11:33:51.309181 19520 consensus_queue.cc:260] T 76811f681a8746d9b89e8ee34ea98ffe P 3b4902991ebe46ae8864959c708e0fe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6e0c033f25422b820518ab00262448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46589 } } peers { permanent_uuid: "00688963452e4d8581a054fb2b5f018f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46827 } } peers { permanent_uuid: "3b4902991ebe46ae8864959c708e0fe1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 } }
I20260926 11:33:51.309386 19519 ts_tablet_manager.cc:1428] T 76811f681a8746d9b89e8ee34ea98ffe P 00688963452e4d8581a054fb2b5f018f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 11:33:51.309451 19239 heartbeater.cc:503] Master 127.0.98.62:43025 was elected leader, sending a full tablet report...
I20260926 11:33:51.310240 19520 ts_tablet_manager.cc:1428] T 76811f681a8746d9b89e8ee34ea98ffe P 3b4902991ebe46ae8864959c708e0fe1: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260926 11:33:51.310628 19501 heartbeater.cc:503] Master 127.0.98.62:43025 was elected leader, sending a full tablet report...
I20260926 11:33:51.323493 19524 raft_consensus.cc:482] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:33:51.323676 19524 raft_consensus.cc:504] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6e0c033f25422b820518ab00262448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46589 } } peers { permanent_uuid: "00688963452e4d8581a054fb2b5f018f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46827 } } peers { permanent_uuid: "3b4902991ebe46ae8864959c708e0fe1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 } }
I20260926 11:33:51.324837 19524 leader_election.cc:288] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00688963452e4d8581a054fb2b5f018f (127.0.98.1:46827), 3b4902991ebe46ae8864959c708e0fe1 (127.0.98.3:39831)
I20260926 11:33:51.328554 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:51.325243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60949 (local address 127.0.98.1:46827)
0926 11:33:51.325397 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:51.325402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:51.325423 (+ 21us) server_negotiation.cc:407] Connection header received
0926 11:33:51.325499 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:51.325505 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:51.325561 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:51.325642 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:51.325931 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:51.326491 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:51.327549 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:51.328005 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:51.328121 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:51.328182 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:51.328360 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 11:33:51.328427 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 11:33:51.328554 19372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:51.325103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46827 (local address 127.0.98.2:60949)
0926 11:33:51.325250 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:51.325267 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:51.325376 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:51.325661 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:51.325664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:51.325680 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:51.325808 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:51.325814 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:51.326617 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:51.326624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:51.327418 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:51.327427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:51.328121 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:51.328125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:51.328227 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:51.328248 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 11:33:51.328345 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 11:33:51.328418 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 11:33:51.328900 19527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:51.325385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39831 (local address 127.0.98.2:50345)
0926 11:33:51.325716 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:51.325729 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:51.325846 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:51.326112 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:51.326115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:51.326131 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:51.326257 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:51.326262 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:51.326980 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:51.326987 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:51.327801 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:51.327810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:51.328435 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:51.328438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:51.328543 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:51.328564 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 11:33:51.328664 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 11:33:51.328750 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":113,"threads_started":1}
I20260926 11:33:51.328997 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:51.325546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50345 (local address 127.0.98.3:39831)
0926 11:33:51.325670 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:51.325674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:51.325813 (+ 139us) server_negotiation.cc:407] Connection header received
0926 11:33:51.325935 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:51.325938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:51.326014 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:51.326100 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:51.326366 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:51.326863 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:51.327926 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:51.328308 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:51.328427 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:51.328480 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:51.328795 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 11:33:51.328862 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 11:33:51.329254 19193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76811f681a8746d9b89e8ee34ea98ffe" candidate_uuid: "ca6e0c033f25422b820518ab00262448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00688963452e4d8581a054fb2b5f018f" is_pre_election: true
I20260926 11:33:51.329577 19455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76811f681a8746d9b89e8ee34ea98ffe" candidate_uuid: "ca6e0c033f25422b820518ab00262448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b4902991ebe46ae8864959c708e0fe1" is_pre_election: true
I20260926 11:33:51.329633 19193 raft_consensus.cc:2454] T 76811f681a8746d9b89e8ee34ea98ffe P 00688963452e4d8581a054fb2b5f018f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca6e0c033f25422b820518ab00262448 in term 0.
I20260926 11:33:51.329885 19455 raft_consensus.cc:2454] T 76811f681a8746d9b89e8ee34ea98ffe P 3b4902991ebe46ae8864959c708e0fe1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca6e0c033f25422b820518ab00262448 in term 0.
I20260926 11:33:51.330174 19259 leader_election.cc:302] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00688963452e4d8581a054fb2b5f018f, ca6e0c033f25422b820518ab00262448; no voters:
I20260926 11:33:51.330472 19524 raft_consensus.cc:2781] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:33:51.330567 19524 raft_consensus.cc:482] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:33:51.330638 19524 raft_consensus.cc:3037] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:33:51.331638 19524 raft_consensus.cc:504] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6e0c033f25422b820518ab00262448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46589 } } peers { permanent_uuid: "00688963452e4d8581a054fb2b5f018f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46827 } } peers { permanent_uuid: "3b4902991ebe46ae8864959c708e0fe1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 } }
I20260926 11:33:51.332001 19524 leader_election.cc:288] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448 [CANDIDATE]: Term 1 election: Requested vote from peers 00688963452e4d8581a054fb2b5f018f (127.0.98.1:46827), 3b4902991ebe46ae8864959c708e0fe1 (127.0.98.3:39831)
I20260926 11:33:51.332398 19193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76811f681a8746d9b89e8ee34ea98ffe" candidate_uuid: "ca6e0c033f25422b820518ab00262448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00688963452e4d8581a054fb2b5f018f"
I20260926 11:33:51.332427 19455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76811f681a8746d9b89e8ee34ea98ffe" candidate_uuid: "ca6e0c033f25422b820518ab00262448" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b4902991ebe46ae8864959c708e0fe1"
I20260926 11:33:51.332537 19193 raft_consensus.cc:3037] T 76811f681a8746d9b89e8ee34ea98ffe P 00688963452e4d8581a054fb2b5f018f [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:33:51.332537 19455 raft_consensus.cc:3037] T 76811f681a8746d9b89e8ee34ea98ffe P 3b4902991ebe46ae8864959c708e0fe1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:33:51.333819 19455 raft_consensus.cc:2454] T 76811f681a8746d9b89e8ee34ea98ffe P 3b4902991ebe46ae8864959c708e0fe1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca6e0c033f25422b820518ab00262448 in term 1.
I20260926 11:33:51.333819 19193 raft_consensus.cc:2454] T 76811f681a8746d9b89e8ee34ea98ffe P 00688963452e4d8581a054fb2b5f018f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca6e0c033f25422b820518ab00262448 in term 1.
I20260926 11:33:51.334143 19259 leader_election.cc:302] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00688963452e4d8581a054fb2b5f018f, ca6e0c033f25422b820518ab00262448; no voters:
W20260926 11:33:51.334146 19240 tablet.cc:2369] T 76811f681a8746d9b89e8ee34ea98ffe P 00688963452e4d8581a054fb2b5f018f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:33:51.334342 19524 raft_consensus.cc:2781] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:33:51.334618 19524 raft_consensus.cc:686] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448 [term 1 LEADER]: Becoming Leader. State: Replica: ca6e0c033f25422b820518ab00262448, State: Running, Role: LEADER
I20260926 11:33:51.334959 19524 consensus_queue.cc:237] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6e0c033f25422b820518ab00262448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46589 } } peers { permanent_uuid: "00688963452e4d8581a054fb2b5f018f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46827 } } peers { permanent_uuid: "3b4902991ebe46ae8864959c708e0fe1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 } }
I20260926 11:33:51.338581 19051 catalog_manager.cc:5495] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca6e0c033f25422b820518ab00262448 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ca6e0c033f25422b820518ab00262448" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca6e0c033f25422b820518ab00262448" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46589 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00688963452e4d8581a054fb2b5f018f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b4902991ebe46ae8864959c708e0fe1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 } health_report { overall_health: UNKNOWN } } }
I20260926 11:33:51.350557 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:51.347447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57834 (local address 127.0.98.2:46589)
0926 11:33:51.347565 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:51.347569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:51.347584 (+ 15us) server_negotiation.cc:407] Connection header received
0926 11:33:51.347644 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:51.347646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:51.347698 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:51.347843 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:33:51.348046 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:51.348535 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:51.349307 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:51.349508 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:51.349581 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:51.349817 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:33:51.350219 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:33:51.350324 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:51.350395 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 11:33:51.350427 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 11:33:51.356660 19193 raft_consensus.cc:1261] T 76811f681a8746d9b89e8ee34ea98ffe P 00688963452e4d8581a054fb2b5f018f [term 1 FOLLOWER]: Refusing update from remote peer ca6e0c033f25422b820518ab00262448: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:33:51.356660 19455 raft_consensus.cc:1261] T 76811f681a8746d9b89e8ee34ea98ffe P 3b4902991ebe46ae8864959c708e0fe1 [term 1 FOLLOWER]: Refusing update from remote peer ca6e0c033f25422b820518ab00262448: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:33:51.357386 19524 consensus_queue.cc:1035] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00688963452e4d8581a054fb2b5f018f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:33:51.357645 19528 consensus_queue.cc:1035] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b4902991ebe46ae8864959c708e0fe1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:33:51.364605 19531 mvcc.cc:205] Tried to move back new op lower bound from 7333570278829096960 to 7333570278750838784. Current Snapshot: MvccSnapshot[applied={T|T < 7333570278829096960}]
I20260926 11:33:51.365470 19532 mvcc.cc:205] Tried to move back new op lower bound from 7333570278829096960 to 7333570278750838784. Current Snapshot: MvccSnapshot[applied={T|T < 7333570278829096960}]
I20260926 11:33:51.367022 19533 mvcc.cc:205] Tried to move back new op lower bound from 7333570278829096960 to 7333570278750838784. Current Snapshot: MvccSnapshot[applied={T|T < 7333570278829096960 or (T in {7333570278829096960})}]
I20260926 11:33:51.374768 19050 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:33:51.374974 19050 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:33:51.377270 19050 catalog_manager.cc:5782] T 00000000000000000000000000000000 P acaa5b6abfe44f7290fe175273594c63: Sending DeleteTablet for 3 replicas of tablet 76811f681a8746d9b89e8ee34ea98ffe
I20260926 11:33:51.378124 19173 tablet_service.cc:1514] Processing DeleteTablet for tablet 76811f681a8746d9b89e8ee34ea98ffe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:33:51 UTC) from {username='slave'} at 127.0.0.1:55126
I20260926 11:33:51.378146 19304 tablet_service.cc:1514] Processing DeleteTablet for tablet 76811f681a8746d9b89e8ee34ea98ffe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:33:51 UTC) from {username='slave'} at 127.0.0.1:57828
I20260926 11:33:51.378659 19540 tablet_replica.cc:330] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448: stopping tablet replica
I20260926 11:33:51.378688 19435 tablet_service.cc:1514] Processing DeleteTablet for tablet 76811f681a8746d9b89e8ee34ea98ffe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:33:51 UTC) from {username='slave'} at 127.0.0.1:39980
I20260926 11:33:51.378971 19540 raft_consensus.cc:2229] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:33:51.379166 19542 tablet_replica.cc:330] T 76811f681a8746d9b89e8ee34ea98ffe P 3b4902991ebe46ae8864959c708e0fe1: stopping tablet replica
I20260926 11:33:51.379454 19540 raft_consensus.cc:2258] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:33:51.379631 19541 tablet_replica.cc:330] T 76811f681a8746d9b89e8ee34ea98ffe P 00688963452e4d8581a054fb2b5f018f: stopping tablet replica
I20260926 11:33:51.379750 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 19111
I20260926 11:33:51.381876 19540 ts_tablet_manager.cc:1905] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:33:51.384191 19540 ts_tablet_manager.cc:1918] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:33:51.384310 19540 log.cc:1196] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-1/wal/wals/76811f681a8746d9b89e8ee34ea98ffe
I20260926 11:33:51.384619 19540 ts_tablet_manager.cc:1939] T 76811f681a8746d9b89e8ee34ea98ffe P ca6e0c033f25422b820518ab00262448: Deleting consensus metadata
I20260926 11:33:51.385722 19038 catalog_manager.cc:4841] TS ca6e0c033f25422b820518ab00262448 (127.0.98.2:46589): tablet 76811f681a8746d9b89e8ee34ea98ffe (table test-table [id=1f4d8676d63f43ff9867b3b87c3a412d]) successfully deleted
W20260926 11:33:51.387852 19037 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46827 (error 108)
I20260926 11:33:51.387990 19542 raft_consensus.cc:2229] T 76811f681a8746d9b89e8ee34ea98ffe P 3b4902991ebe46ae8864959c708e0fe1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:33:51.388375 19542 raft_consensus.cc:2258] T 76811f681a8746d9b89e8ee34ea98ffe P 3b4902991ebe46ae8864959c708e0fe1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:33:51.388753 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 19242
I20260926 11:33:51.389029 19512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:51.388680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46827 (local address 127.0.0.1:55128)
0926 11:33:51.388827 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:51.388913 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 11:33:51.389458 19037 catalog_manager.cc:4584] TS 00688963452e4d8581a054fb2b5f018f (127.0.98.1:46827): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 76811f681a8746d9b89e8ee34ea98ffe: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46827: connect: Connection refused (error 111)
I20260926 11:33:51.391196 19542 ts_tablet_manager.cc:1905] T 76811f681a8746d9b89e8ee34ea98ffe P 3b4902991ebe46ae8864959c708e0fe1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:33:51.394711 19542 ts_tablet_manager.cc:1918] T 76811f681a8746d9b89e8ee34ea98ffe P 3b4902991ebe46ae8864959c708e0fe1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:33:51.394819 19542 log.cc:1196] T 76811f681a8746d9b89e8ee34ea98ffe P 3b4902991ebe46ae8864959c708e0fe1: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790422180210734-392-0/minicluster-data/ts-2/wal/wals/76811f681a8746d9b89e8ee34ea98ffe
I20260926 11:33:51.395167 19542 ts_tablet_manager.cc:1939] T 76811f681a8746d9b89e8ee34ea98ffe P 3b4902991ebe46ae8864959c708e0fe1: Deleting consensus metadata
I20260926 11:33:51.396162 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 19373
I20260926 11:33:51.401885 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 19020
2026-09-26T11:33:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1027 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:33:51 dist-test-slave-mrck krb5kdc[19546](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:33:51 dist-test-slave-mrck krb5kdc[19546](info): set up 2 sockets
Sep 26 11:33:51 dist-test-slave-mrck krb5kdc[19546](info): commencing operation
krb5kdc: starting...
W20260926 11:33:52.427919 392 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:33:52 dist-test-slave-mrck krb5kdc[19546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:33:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:33:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:33:52.546206 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37857
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:33:52.654584 19559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:52.654899 19559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:52.654969 19559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:52.658983 19559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:33:52.659078 19559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:52.659118 19559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:52.659176 19559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:33:52.659216 19559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:33:52.665139 19559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:52.666194 19559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:52.667233 19559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:33:52.675935 19564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:52.675928 19567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:52.676095 19559 server_base.cc:1034] running on GCE node
W20260926 11:33:52.675928 19565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:52.676776 19559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:52.677932 19559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:52.679119 19559 hybrid_clock.cc:648] HybridClock initialized: now 1790422432679105 us; error 28 us; skew 500 ppm
Sep 26 11:33:52 dist-test-slave-mrck krb5kdc[19546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:33:52.682056 19559 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:33:52.682919 19559 webserver.cc:460] Webserver started at http://127.0.98.62:42197/ using document root <none> and password file <none>
I20260926 11:33:52.683444 19559 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:52.683525 19559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:52.683697 19559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:33:52.685778 19559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "379f0f478d61496eac7380f7c947e2ef"
format_stamp: "Formatted at 2026-09-26 11:33:52 on dist-test-slave-mrck"
server_key: "c0d44f532bffb72e60f79624089886fe"
server_key_iv: "8755684de9411e78986ba449b39cd22e"
server_key_version: "encryptionkey@0"
I20260926 11:33:52.686316 19559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "379f0f478d61496eac7380f7c947e2ef"
format_stamp: "Formatted at 2026-09-26 11:33:52 on dist-test-slave-mrck"
server_key: "c0d44f532bffb72e60f79624089886fe"
server_key_iv: "8755684de9411e78986ba449b39cd22e"
server_key_version: "encryptionkey@0"
I20260926 11:33:52.690106 19559 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 11:33:52.692684 19574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:52.693992 19559 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:33:52.694144 19559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "379f0f478d61496eac7380f7c947e2ef"
format_stamp: "Formatted at 2026-09-26 11:33:52 on dist-test-slave-mrck"
server_key: "c0d44f532bffb72e60f79624089886fe"
server_key_iv: "8755684de9411e78986ba449b39cd22e"
server_key_version: "encryptionkey@0"
I20260926 11:33:52.694265 19559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:52.720490 19559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:52.723609 19559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:52.723883 19559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:52.731323 19626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37857 every 8 connection(s)
I20260926 11:33:52.731326 19559 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37857
I20260926 11:33:52.732386 19559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:33:52.735054 19627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:33:52.740329 19627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 379f0f478d61496eac7380f7c947e2ef: Bootstrap starting.
I20260926 11:33:52.742369 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 19559
I20260926 11:33:52.742553 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:33:52.742754 19627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 379f0f478d61496eac7380f7c947e2ef: Neither blocks nor log segments found. Creating new log.
I20260926 11:33:52.742916 392 external_mini_cluster.cc:1428] Setting key eafe657901d59d044addbc0e22b2acd4
I20260926 11:33:52.743494 19627 log.cc:824] T 00000000000000000000000000000000 P 379f0f478d61496eac7380f7c947e2ef: Log is configured to *not* fsync() on all Append() calls
I20260926 11:33:52.745859 19627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 379f0f478d61496eac7380f7c947e2ef: No bootstrap required, opened a new log
I20260926 11:33:52.750168 19627 raft_consensus.cc:348] T 00000000000000000000000000000000 P 379f0f478d61496eac7380f7c947e2ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "379f0f478d61496eac7380f7c947e2ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37857 } }
I20260926 11:33:52.750427 19627 raft_consensus.cc:374] T 00000000000000000000000000000000 P 379f0f478d61496eac7380f7c947e2ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:33:52.750520 19627 raft_consensus.cc:729] T 00000000000000000000000000000000 P 379f0f478d61496eac7380f7c947e2ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 379f0f478d61496eac7380f7c947e2ef, State: Initialized, Role: FOLLOWER
I20260926 11:33:52.750995 19627 consensus_queue.cc:260] T 00000000000000000000000000000000 P 379f0f478d61496eac7380f7c947e2ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "379f0f478d61496eac7380f7c947e2ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37857 } }
I20260926 11:33:52.751155 19627 raft_consensus.cc:388] T 00000000000000000000000000000000 P 379f0f478d61496eac7380f7c947e2ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:33:52.751226 19627 raft_consensus.cc:482] T 00000000000000000000000000000000 P 379f0f478d61496eac7380f7c947e2ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:33:52.751322 19627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 379f0f478d61496eac7380f7c947e2ef [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:33:52.752442 19627 raft_consensus.cc:504] T 00000000000000000000000000000000 P 379f0f478d61496eac7380f7c947e2ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "379f0f478d61496eac7380f7c947e2ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37857 } }
Sep 26 11:33:52 dist-test-slave-mrck krb5kdc[19546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:33:52.752805 19627 leader_election.cc:302] T 00000000000000000000000000000000 P 379f0f478d61496eac7380f7c947e2ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 379f0f478d61496eac7380f7c947e2ef; no voters:
I20260926 11:33:52.753180 19627 leader_election.cc:288] T 00000000000000000000000000000000 P 379f0f478d61496eac7380f7c947e2ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:33:52.753296 19632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 379f0f478d61496eac7380f7c947e2ef [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:33:52.753542 19632 raft_consensus.cc:686] T 00000000000000000000000000000000 P 379f0f478d61496eac7380f7c947e2ef [term 1 LEADER]: Becoming Leader. State: Replica: 379f0f478d61496eac7380f7c947e2ef, State: Running, Role: LEADER
I20260926 11:33:52.753965 19632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 379f0f478d61496eac7380f7c947e2ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "379f0f478d61496eac7380f7c947e2ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37857 } }
I20260926 11:33:52.754580 19627 sys_catalog.cc:564] T 00000000000000000000000000000000 P 379f0f478d61496eac7380f7c947e2ef [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:33:52.756456 19633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 379f0f478d61496eac7380f7c947e2ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 379f0f478d61496eac7380f7c947e2ef. Latest consensus state: current_term: 1 leader_uuid: "379f0f478d61496eac7380f7c947e2ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "379f0f478d61496eac7380f7c947e2ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37857 } } }
I20260926 11:33:52.756474 19634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 379f0f478d61496eac7380f7c947e2ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "379f0f478d61496eac7380f7c947e2ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "379f0f478d61496eac7380f7c947e2ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37857 } } }
I20260926 11:33:52.756711 19633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 379f0f478d61496eac7380f7c947e2ef [sys.catalog]: This master's current role is: LEADER
I20260926 11:33:52.756718 19634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 379f0f478d61496eac7380f7c947e2ef [sys.catalog]: This master's current role is: LEADER
I20260926 11:33:52.757148 19647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:33:52.759256 19631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:52.744132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54566 (local address 127.0.98.62:37857)
0926 11:33:52.744598 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:52.744618 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:52.744657 (+ 39us) server_negotiation.cc:407] Connection header received
0926 11:33:52.745468 (+ 811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:52.745491 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:52.745870 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:52.746224 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:52.746903 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:52.748464 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:52.749363 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:52.750050 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:52.750213 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:52.753298 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:52.753305 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:52.753323 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:33:52.753358 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:52.755502 (+ 2144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:52.756152 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:52.756160 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:52.756169 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:52.756334 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:52.756674 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:52.756680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:52.756683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:52.757211 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:52.757428 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:52.758146 (+ 718us) server_negotiation.cc:299] Negotiation successful
0926 11:33:52.758383 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":122,"threads_started":1}
I20260926 11:33:52.760565 19647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:33:52.765885 19647 catalog_manager.cc:1347] Generated new cluster ID: 3311742b65a4419c94d1cd09ebbf61c0
I20260926 11:33:52.765976 19647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:33:52.788794 19647 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:33:52.790145 19647 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:33:52.803133 19647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 379f0f478d61496eac7380f7c947e2ef: Generated new TSK 0
I20260926 11:33:52.804031 19647 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 11:33:52.834056 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37857
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:33:52.946161 19653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:52.946410 19653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:52.946455 19653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:52.950498 19653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:52.950552 19653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:52.950609 19653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:33:52.955886 19653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:52.956959 19653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:52.957890 19653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:33:52.967463 19658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:52.967470 19659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:52.967617 19653 server_base.cc:1034] running on GCE node
W20260926 11:33:52.967480 19661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:52.968122 19653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:52.968740 19653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:52.969938 19653 hybrid_clock.cc:648] HybridClock initialized: now 1790422432969924 us; error 27 us; skew 500 ppm
Sep 26 11:33:52 dist-test-slave-mrck krb5kdc[19546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:33:52.972565 19653 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 11:33:52.973428 19653 webserver.cc:460] Webserver started at http://127.0.98.1:46397/ using document root <none> and password file <none>
I20260926 11:33:52.973977 19653 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:52.974068 19653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:52.974290 19653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:33:52.976325 19653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-0/data/instance:
uuid: "77fe14a61a29437e93296b08f8a8cff4"
format_stamp: "Formatted at 2026-09-26 11:33:52 on dist-test-slave-mrck"
server_key: "b6e7d9c9545d12955a59a1d408a98876"
server_key_iv: "7366be2dd002c8e9c0b58fe7c4235ee4"
server_key_version: "encryptionkey@0"
I20260926 11:33:52.976866 19653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "77fe14a61a29437e93296b08f8a8cff4"
format_stamp: "Formatted at 2026-09-26 11:33:52 on dist-test-slave-mrck"
server_key: "b6e7d9c9545d12955a59a1d408a98876"
server_key_iv: "7366be2dd002c8e9c0b58fe7c4235ee4"
server_key_version: "encryptionkey@0"
I20260926 11:33:52.980332 19653 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 11:33:52.982736 19668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:52.983955 19653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260926 11:33:52.984081 19653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "77fe14a61a29437e93296b08f8a8cff4"
format_stamp: "Formatted at 2026-09-26 11:33:52 on dist-test-slave-mrck"
server_key: "b6e7d9c9545d12955a59a1d408a98876"
server_key_iv: "7366be2dd002c8e9c0b58fe7c4235ee4"
server_key_version: "encryptionkey@0"
I20260926 11:33:52.984187 19653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:52.993253 19653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:52.995957 19653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:52.996146 19653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:52.996688 19653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:33:52.997471 19653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:33:52.997539 19653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:52.997617 19653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:33:52.997669 19653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:53.006139 19653 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40783
I20260926 11:33:53.006165 19781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40783 every 8 connection(s)
I20260926 11:33:53.007103 19653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:33:53.010198 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 19653
I20260926 11:33:53.010342 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:33:53.010649 392 external_mini_cluster.cc:1428] Setting key 9ccdf3e37e7738bf70738bfe2283a25c
Sep 26 11:33:53 dist-test-slave-mrck krb5kdc[19546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 11:33:53.021230 19631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:53.008993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34053 (local address 127.0.98.62:37857)
0926 11:33:53.009150 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:53.009155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:53.010089 (+ 934us) server_negotiation.cc:407] Connection header received
0926 11:33:53.011256 (+ 1167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:53.011258 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:53.011305 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:53.011401 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:53.012745 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.013322 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.014665 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.014885 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.014970 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:53.017580 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:53.017584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:53.017587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:33:53.017612 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:53.019148 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:53.019650 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:53.019654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:53.019656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:53.019780 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:53.020135 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:53.020141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:53.020145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:53.020330 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:53.020432 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:53.020924 (+ 492us) server_negotiation.cc:299] Negotiation successful
0926 11:33:53.021058 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 11:33:53.021993 19784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:53.009422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37857 (local address 127.0.98.1:34053)
0926 11:33:53.009875 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:53.009948 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:53.011012 (+ 1064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:53.011581 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:53.011588 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:53.012107 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:33:53.012592 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:53.012605 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.013479 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.013482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:53.014547 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:53.014560 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.015033 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.015040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:53.015266 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:53.015988 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:33:53.016003 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:33:53.017412 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:33:53.019264 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:33:53.019269 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:33:53.019284 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:33:53.019546 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:33:53.019880 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:33:53.019884 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:33:53.019886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:33:53.020038 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:33:53.020450 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:33:53.020457 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:33:53.020698 (+ 241us) client_negotiation.cc:769] Sending connection context
0926 11:33:53.020886 (+ 188us) client_negotiation.cc:240] Negotiation successful
0926 11:33:53.021202 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 11:33:53.023253 19782 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37857
I20260926 11:33:53.023491 19782 heartbeater.cc:464] Registering TS with master...
I20260926 11:33:53.024113 19782 heartbeater.cc:511] Master 127.0.98.62:37857 requested a full tablet report, sending...
I20260926 11:33:53.025707 19591 ts_manager.cc:194] Registered new tserver with Master: 77fe14a61a29437e93296b08f8a8cff4 (127.0.98.1:40783)
I20260926 11:33:53.026893 19591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34053
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 11:33:53.038164 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37857
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:33:53.147087 19787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:53.147368 19787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:53.147413 19787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:53.151394 19787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:53.151454 19787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:53.151554 19787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:33:53.157045 19787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:53.158097 19787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:53.159116 19787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:33:53.168452 19793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:53.168534 19787 server_base.cc:1034] running on GCE node
W20260926 11:33:53.168462 19795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:53.168519 19792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:53.169101 19787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:53.169708 19787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:53.170900 19787 hybrid_clock.cc:648] HybridClock initialized: now 1790422433170886 us; error 24 us; skew 500 ppm
Sep 26 11:33:53 dist-test-slave-mrck krb5kdc[19546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:33:53.173558 19787 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 11:33:53.174439 19787 webserver.cc:460] Webserver started at http://127.0.98.2:45339/ using document root <none> and password file <none>
I20260926 11:33:53.175004 19787 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:53.175091 19787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:53.175323 19787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:33:53.177379 19787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-1/data/instance:
uuid: "1a432b3c1a5e4c4f9dd195a2b4efcfbc"
format_stamp: "Formatted at 2026-09-26 11:33:53 on dist-test-slave-mrck"
server_key: "9c909fabf2d7d9336b2a39df7fe25fad"
server_key_iv: "923aedd09fdcb2a9e18a146c7c83ee05"
server_key_version: "encryptionkey@0"
I20260926 11:33:53.177935 19787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1a432b3c1a5e4c4f9dd195a2b4efcfbc"
format_stamp: "Formatted at 2026-09-26 11:33:53 on dist-test-slave-mrck"
server_key: "9c909fabf2d7d9336b2a39df7fe25fad"
server_key_iv: "923aedd09fdcb2a9e18a146c7c83ee05"
server_key_version: "encryptionkey@0"
I20260926 11:33:53.181506 19787 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 11:33:53.183897 19802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:53.185082 19787 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:33:53.185171 19787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "1a432b3c1a5e4c4f9dd195a2b4efcfbc"
format_stamp: "Formatted at 2026-09-26 11:33:53 on dist-test-slave-mrck"
server_key: "9c909fabf2d7d9336b2a39df7fe25fad"
server_key_iv: "923aedd09fdcb2a9e18a146c7c83ee05"
server_key_version: "encryptionkey@0"
I20260926 11:33:53.185253 19787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:53.205701 19787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:53.208554 19787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:53.208709 19787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:53.209254 19787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:33:53.210016 19787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:33:53.210062 19787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:53.210099 19787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:33:53.210122 19787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:53.218969 19787 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42151
I20260926 11:33:53.218988 19915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42151 every 8 connection(s)
I20260926 11:33:53.219868 19787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:33:53.224392 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 19787
I20260926 11:33:53.224527 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:33:53.224797 392 external_mini_cluster.cc:1428] Setting key b6bab581d8fdf319410013f555c87587
Sep 26 11:33:53 dist-test-slave-mrck krb5kdc[19546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 11:33:53.233832 19631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:53.221643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55103 (local address 127.0.98.62:37857)
0926 11:33:53.221776 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:53.221779 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:53.222656 (+ 877us) server_negotiation.cc:407] Connection header received
0926 11:33:53.223584 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:53.223587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:53.223647 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:53.223747 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:53.225082 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.225628 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.227174 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.227402 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.227487 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:53.230226 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:53.230230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:53.230233 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:33:53.230274 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:53.231827 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:53.232330 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:53.232335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:53.232337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:53.232424 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:53.232768 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:53.232771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:53.232773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:53.232956 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:53.233058 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:53.233551 (+ 493us) server_negotiation.cc:299] Negotiation successful
0926 11:33:53.233673 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 11:33:53.234668 19918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:53.222027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37857 (local address 127.0.98.2:55103)
0926 11:33:53.222507 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:53.222560 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:53.223351 (+ 791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:53.223950 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:53.223957 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:53.224309 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:33:53.224938 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:53.224956 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.225759 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.225762 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:53.227043 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:53.227057 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.227556 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.227563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:53.227833 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:53.228544 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:33:53.228561 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:33:53.230059 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:33:53.231949 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:33:53.231955 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:33:53.231971 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:33:53.232225 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:33:53.232516 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:33:53.232521 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:33:53.232523 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:33:53.232666 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:33:53.233068 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:33:53.233076 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:33:53.233322 (+ 246us) client_negotiation.cc:769] Sending connection context
0926 11:33:53.233513 (+ 191us) client_negotiation.cc:240] Negotiation successful
0926 11:33:53.233832 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":108,"threads_started":1}
I20260926 11:33:53.235934 19916 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37857
I20260926 11:33:53.236187 19916 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 11:33:53.236783 19916 heartbeater.cc:511] Master 127.0.98.62:37857 requested a full tablet report, sending...
I20260926 11:33:53.237977 19591 ts_manager.cc:194] Registered new tserver with Master: 1a432b3c1a5e4c4f9dd195a2b4efcfbc (127.0.98.2:42151)
I20260926 11:33:53.238566 19591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 11:33:53.250500 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37857
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:33:53.362995 19921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:53.363252 19921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:53.363296 19921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:53.367384 19921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:53.367446 19921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:53.367508 19921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:33:53.372689 19921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:53.373677 19921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:53.374715 19921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 11:33:53.384300 19921 server_base.cc:1034] running on GCE node
W20260926 11:33:53.384205 19927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:53.384232 19926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:53.384196 19929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:53.384958 19921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:53.385564 19921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:53.386745 19921 hybrid_clock.cc:648] HybridClock initialized: now 1790422433386726 us; error 29 us; skew 500 ppm
Sep 26 11:33:53 dist-test-slave-mrck krb5kdc[19546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:33:53.389904 19921 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 11:33:53.390831 19921 webserver.cc:460] Webserver started at http://127.0.98.3:42383/ using document root <none> and password file <none>
I20260926 11:33:53.391422 19921 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:53.391486 19921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:53.391705 19921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:33:53.393813 19921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-2/data/instance:
uuid: "b5e4dead5cd145b3a2ce627c1bb4df0f"
format_stamp: "Formatted at 2026-09-26 11:33:53 on dist-test-slave-mrck"
server_key: "27d8273f7b4dd78ac9153847dfba7044"
server_key_iv: "200c7c2255429483b872cae828ed403d"
server_key_version: "encryptionkey@0"
I20260926 11:33:53.394363 19921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b5e4dead5cd145b3a2ce627c1bb4df0f"
format_stamp: "Formatted at 2026-09-26 11:33:53 on dist-test-slave-mrck"
server_key: "27d8273f7b4dd78ac9153847dfba7044"
server_key_iv: "200c7c2255429483b872cae828ed403d"
server_key_version: "encryptionkey@0"
I20260926 11:33:53.398105 19921 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260926 11:33:53.400622 19936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:53.401793 19921 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260926 11:33:53.401934 19921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "b5e4dead5cd145b3a2ce627c1bb4df0f"
format_stamp: "Formatted at 2026-09-26 11:33:53 on dist-test-slave-mrck"
server_key: "27d8273f7b4dd78ac9153847dfba7044"
server_key_iv: "200c7c2255429483b872cae828ed403d"
server_key_version: "encryptionkey@0"
I20260926 11:33:53.402034 19921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:53.415223 19921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:53.418238 19921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:53.418406 19921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:53.418927 19921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:33:53.419718 19921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:33:53.419826 19921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:53.419869 19921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:33:53.419890 19921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:53.428493 19921 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42313
I20260926 11:33:53.428575 20049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42313 every 8 connection(s)
I20260926 11:33:53.429378 19921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:33:53.436967 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 19921
I20260926 11:33:53.437129 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:33:53.437449 392 external_mini_cluster.cc:1428] Setting key 0df20d155167fda0e33f126df5905a6e
Sep 26 11:33:53 dist-test-slave-mrck krb5kdc[19546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422433, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:33:53.442509 19631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:53.431198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53383 (local address 127.0.98.62:37857)
0926 11:33:53.431337 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:53.431340 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:53.432251 (+ 911us) server_negotiation.cc:407] Connection header received
0926 11:33:53.433145 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:53.433149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:53.433210 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:53.433315 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:53.434477 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.435009 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.436234 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.436464 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.436577 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:53.439221 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:53.439225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:53.439227 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:33:53.439256 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:53.440807 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:53.441266 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:53.441269 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:53.441270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:53.441342 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:53.441621 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:53.441623 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:53.441624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:53.441785 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:53.441857 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:53.442299 (+ 442us) server_negotiation.cc:299] Negotiation successful
0926 11:33:53.442393 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 11:33:53.443334 20052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:53.431588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37857 (local address 127.0.98.3:53383)
0926 11:33:53.432105 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:53.432146 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:53.432928 (+ 782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:53.433488 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:53.433495 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:53.433848 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:33:53.434328 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:53.434340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.435132 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.435136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:53.436096 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:53.436106 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.436614 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.436618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:53.436786 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:53.437418 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:33:53.437434 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:33:53.439054 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:33:53.440909 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:33:53.440914 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:33:53.440926 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:33:53.441182 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:33:53.441418 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:33:53.441420 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:33:53.441421 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:33:53.441547 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:33:53.441865 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:33:53.441870 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:33:53.442104 (+ 234us) client_negotiation.cc:769] Sending connection context
0926 11:33:53.442255 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 11:33:53.442502 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":172,"threads_started":1}
I20260926 11:33:53.444501 20050 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37857
I20260926 11:33:53.444733 20050 heartbeater.cc:464] Registering TS with master...
I20260926 11:33:53.445210 20050 heartbeater.cc:511] Master 127.0.98.62:37857 requested a full tablet report, sending...
I20260926 11:33:53.446213 19591 ts_manager.cc:194] Registered new tserver with Master: b5e4dead5cd145b3a2ce627c1bb4df0f (127.0.98.3:42313)
I20260926 11:33:53.446805 19591 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53383
I20260926 11:33:53.451363 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:33:53.460920 19631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:53.453385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54574 (local address 127.0.98.62:37857)
0926 11:33:53.453510 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:53.453513 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:53.453798 (+ 285us) server_negotiation.cc:407] Connection header received
0926 11:33:53.453969 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:53.453972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:53.454007 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:53.454073 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:53.454846 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.455477 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.456645 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.456841 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.456924 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:53.458284 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:53.458286 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:53.458288 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:33:53.458304 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:53.459515 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:53.459964 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:53.459966 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:53.459968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:53.460038 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:53.460237 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:53.460239 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:53.460240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:53.460421 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:53.460537 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:53.460713 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 11:33:53.460802 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 11:33:53.463878 19591 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:33:53.466902 19591 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:33:53.481309 20061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:53.475347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40783 (local address 127.0.0.1:60014)
0926 11:33:53.475688 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:53.475796 (+ 108us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:53.475927 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:53.476509 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:53.476516 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:53.476541 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:53.476689 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:53.476696 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.478078 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.478081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:53.479727 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:53.479761 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.480752 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.480755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:53.480886 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:53.480929 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 11:33:53.481038 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 11:33:53.481149 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":74,"spinlock_wait_cycles":7936,"thread_start_us":117,"threads_started":1}
I20260926 11:33:53.481331 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:53.475557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60014 (local address 127.0.98.1:40783)
0926 11:33:53.475954 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:53.475967 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:53.475993 (+ 26us) server_negotiation.cc:407] Connection header received
0926 11:33:53.476071 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:53.476076 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:53.476315 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:53.476474 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:53.476827 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.477963 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.479883 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.480584 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.480695 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:53.480847 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:53.481085 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 11:33:53.481179 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":121,"threads_started":1}
I20260926 11:33:53.481992 20063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:53.476237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42313 (local address 127.0.0.1:39414)
0926 11:33:53.476485 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:53.476501 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:53.476588 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:53.478000 (+ 1412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:53.478004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:53.478025 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:53.478144 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:53.478149 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.479176 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.479183 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:53.480477 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:53.480487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.481574 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.481578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:53.481672 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:53.481693 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 11:33:53.481784 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 11:33:53.481853 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":3,"thread_start_us":121,"threads_started":1}
I20260926 11:33:53.482030 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:53.476915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39414 (local address 127.0.98.3:42313)
0926 11:33:53.477553 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:53.477564 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:53.477585 (+ 21us) server_negotiation.cc:407] Connection header received
0926 11:33:53.477656 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:53.477661 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:53.477836 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:53.478255 (+ 419us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:53.478377 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.479047 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.480610 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.481416 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.481571 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:53.481712 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:53.481808 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 11:33:53.481889 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":149,"threads_started":1}
I20260926 11:33:53.482220 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:53.476571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42151 (local address 127.0.0.1:38234)
0926 11:33:53.477347 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:53.477361 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:53.477461 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:53.477852 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:53.477855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:53.477873 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:53.477978 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:53.477985 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.479373 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.479377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:53.480583 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:53.480593 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.481813 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.481816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:53.481903 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:53.481927 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 11:33:53.481995 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 11:33:53.482072 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"mutex_wait_us":2,"thread_start_us":83,"threads_started":1}
I20260926 11:33:53.482623 20065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:53.476880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38234 (local address 127.0.98.2:42151)
0926 11:33:53.477364 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:53.477380 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:53.477423 (+ 43us) server_negotiation.cc:407] Connection header received
0926 11:33:53.477568 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:53.477579 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:53.477719 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:53.477868 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:53.478104 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.479239 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.480702 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.481536 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.481630 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:53.481751 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:53.482396 (+ 645us) server_negotiation.cc:299] Negotiation successful
0926 11:33:53.482486 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":110,"threads_started":1}
I20260926 11:33:53.484931 19984 tablet_service.cc:1467] Processing CreateTablet for tablet 67dc72125e894703aa54ebf373f71370 (DEFAULT_TABLE table=test-table [id=1eab97f6dc114e05b6edeed988bb1c65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:33:53.485304 19850 tablet_service.cc:1467] Processing CreateTablet for tablet 67dc72125e894703aa54ebf373f71370 (DEFAULT_TABLE table=test-table [id=1eab97f6dc114e05b6edeed988bb1c65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:33:53.485304 19716 tablet_service.cc:1467] Processing CreateTablet for tablet 67dc72125e894703aa54ebf373f71370 (DEFAULT_TABLE table=test-table [id=1eab97f6dc114e05b6edeed988bb1c65]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:33:53.485968 19984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67dc72125e894703aa54ebf373f71370. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:33:53.486312 19850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67dc72125e894703aa54ebf373f71370. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:33:53.486312 19716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67dc72125e894703aa54ebf373f71370. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:33:53.492178 20067 tablet_bootstrap.cc:492] T 67dc72125e894703aa54ebf373f71370 P b5e4dead5cd145b3a2ce627c1bb4df0f: Bootstrap starting.
I20260926 11:33:53.493659 20069 tablet_bootstrap.cc:492] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc: Bootstrap starting.
I20260926 11:33:53.495538 20067 tablet_bootstrap.cc:654] T 67dc72125e894703aa54ebf373f71370 P b5e4dead5cd145b3a2ce627c1bb4df0f: Neither blocks nor log segments found. Creating new log.
I20260926 11:33:53.495868 20068 tablet_bootstrap.cc:492] T 67dc72125e894703aa54ebf373f71370 P 77fe14a61a29437e93296b08f8a8cff4: Bootstrap starting.
I20260926 11:33:53.495990 20069 tablet_bootstrap.cc:654] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc: Neither blocks nor log segments found. Creating new log.
I20260926 11:33:53.496831 20069 log.cc:824] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc: Log is configured to *not* fsync() on all Append() calls
I20260926 11:33:53.496831 20067 log.cc:824] T 67dc72125e894703aa54ebf373f71370 P b5e4dead5cd145b3a2ce627c1bb4df0f: Log is configured to *not* fsync() on all Append() calls
I20260926 11:33:53.497890 20068 tablet_bootstrap.cc:654] T 67dc72125e894703aa54ebf373f71370 P 77fe14a61a29437e93296b08f8a8cff4: Neither blocks nor log segments found. Creating new log.
I20260926 11:33:53.498889 20067 tablet_bootstrap.cc:492] T 67dc72125e894703aa54ebf373f71370 P b5e4dead5cd145b3a2ce627c1bb4df0f: No bootstrap required, opened a new log
I20260926 11:33:53.498879 20069 tablet_bootstrap.cc:492] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc: No bootstrap required, opened a new log
I20260926 11:33:53.499073 20069 ts_tablet_manager.cc:1397] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 11:33:53.499074 20067 ts_tablet_manager.cc:1397] T 67dc72125e894703aa54ebf373f71370 P b5e4dead5cd145b3a2ce627c1bb4df0f: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260926 11:33:53.499145 20068 log.cc:824] T 67dc72125e894703aa54ebf373f71370 P 77fe14a61a29437e93296b08f8a8cff4: Log is configured to *not* fsync() on all Append() calls
I20260926 11:33:53.500830 20068 tablet_bootstrap.cc:492] T 67dc72125e894703aa54ebf373f71370 P 77fe14a61a29437e93296b08f8a8cff4: No bootstrap required, opened a new log
I20260926 11:33:53.501034 20068 ts_tablet_manager.cc:1397] T 67dc72125e894703aa54ebf373f71370 P 77fe14a61a29437e93296b08f8a8cff4: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260926 11:33:53.503033 20067 raft_consensus.cc:348] T 67dc72125e894703aa54ebf373f71370 P b5e4dead5cd145b3a2ce627c1bb4df0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe14a61a29437e93296b08f8a8cff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40783 } } peers { permanent_uuid: "b5e4dead5cd145b3a2ce627c1bb4df0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42313 } } peers { permanent_uuid: "1a432b3c1a5e4c4f9dd195a2b4efcfbc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42151 } }
I20260926 11:33:53.503033 20069 raft_consensus.cc:348] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe14a61a29437e93296b08f8a8cff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40783 } } peers { permanent_uuid: "b5e4dead5cd145b3a2ce627c1bb4df0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42313 } } peers { permanent_uuid: "1a432b3c1a5e4c4f9dd195a2b4efcfbc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42151 } }
I20260926 11:33:53.503325 20069 raft_consensus.cc:374] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:33:53.503325 20067 raft_consensus.cc:374] T 67dc72125e894703aa54ebf373f71370 P b5e4dead5cd145b3a2ce627c1bb4df0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:33:53.503396 20067 raft_consensus.cc:729] T 67dc72125e894703aa54ebf373f71370 P b5e4dead5cd145b3a2ce627c1bb4df0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5e4dead5cd145b3a2ce627c1bb4df0f, State: Initialized, Role: FOLLOWER
I20260926 11:33:53.503435 20069 raft_consensus.cc:729] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a432b3c1a5e4c4f9dd195a2b4efcfbc, State: Initialized, Role: FOLLOWER
I20260926 11:33:53.503918 20067 consensus_queue.cc:260] T 67dc72125e894703aa54ebf373f71370 P b5e4dead5cd145b3a2ce627c1bb4df0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe14a61a29437e93296b08f8a8cff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40783 } } peers { permanent_uuid: "b5e4dead5cd145b3a2ce627c1bb4df0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42313 } } peers { permanent_uuid: "1a432b3c1a5e4c4f9dd195a2b4efcfbc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42151 } }
I20260926 11:33:53.503918 20069 consensus_queue.cc:260] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe14a61a29437e93296b08f8a8cff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40783 } } peers { permanent_uuid: "b5e4dead5cd145b3a2ce627c1bb4df0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42313 } } peers { permanent_uuid: "1a432b3c1a5e4c4f9dd195a2b4efcfbc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42151 } }
I20260926 11:33:53.503908 20068 raft_consensus.cc:348] T 67dc72125e894703aa54ebf373f71370 P 77fe14a61a29437e93296b08f8a8cff4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe14a61a29437e93296b08f8a8cff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40783 } } peers { permanent_uuid: "b5e4dead5cd145b3a2ce627c1bb4df0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42313 } } peers { permanent_uuid: "1a432b3c1a5e4c4f9dd195a2b4efcfbc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42151 } }
I20260926 11:33:53.504163 20068 raft_consensus.cc:374] T 67dc72125e894703aa54ebf373f71370 P 77fe14a61a29437e93296b08f8a8cff4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:33:53.504220 20068 raft_consensus.cc:729] T 67dc72125e894703aa54ebf373f71370 P 77fe14a61a29437e93296b08f8a8cff4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77fe14a61a29437e93296b08f8a8cff4, State: Initialized, Role: FOLLOWER
I20260926 11:33:53.504628 20068 consensus_queue.cc:260] T 67dc72125e894703aa54ebf373f71370 P 77fe14a61a29437e93296b08f8a8cff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe14a61a29437e93296b08f8a8cff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40783 } } peers { permanent_uuid: "b5e4dead5cd145b3a2ce627c1bb4df0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42313 } } peers { permanent_uuid: "1a432b3c1a5e4c4f9dd195a2b4efcfbc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42151 } }
I20260926 11:33:53.504959 20069 ts_tablet_manager.cc:1428] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 11:33:53.504959 20067 ts_tablet_manager.cc:1428] T 67dc72125e894703aa54ebf373f71370 P b5e4dead5cd145b3a2ce627c1bb4df0f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 11:33:53.505297 19916 heartbeater.cc:503] Master 127.0.98.62:37857 was elected leader, sending a full tablet report...
I20260926 11:33:53.505297 20050 heartbeater.cc:503] Master 127.0.98.62:37857 was elected leader, sending a full tablet report...
I20260926 11:33:53.505589 20068 ts_tablet_manager.cc:1428] T 67dc72125e894703aa54ebf373f71370 P 77fe14a61a29437e93296b08f8a8cff4: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260926 11:33:53.506019 19782 heartbeater.cc:503] Master 127.0.98.62:37857 was elected leader, sending a full tablet report...
W20260926 11:33:53.508323 19783 tablet.cc:2369] T 67dc72125e894703aa54ebf373f71370 P 77fe14a61a29437e93296b08f8a8cff4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:33:53.575047 20073 raft_consensus.cc:482] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:33:53.575279 20073 raft_consensus.cc:504] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe14a61a29437e93296b08f8a8cff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40783 } } peers { permanent_uuid: "b5e4dead5cd145b3a2ce627c1bb4df0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42313 } } peers { permanent_uuid: "1a432b3c1a5e4c4f9dd195a2b4efcfbc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42151 } }
I20260926 11:33:53.576507 20073 leader_election.cc:288] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77fe14a61a29437e93296b08f8a8cff4 (127.0.98.1:40783), b5e4dead5cd145b3a2ce627c1bb4df0f (127.0.98.3:42313)
I20260926 11:33:53.581251 20075 raft_consensus.cc:482] T 67dc72125e894703aa54ebf373f71370 P 77fe14a61a29437e93296b08f8a8cff4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:33:53.581460 19918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:53.576800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40783 (local address 127.0.98.2:39629)
0926 11:33:53.576963 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:53.576979 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:53.577104 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:53.577428 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:53.577431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:53.577453 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:53.577596 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:53.577607 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.578628 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.578630 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:53.579994 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:53.580000 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.580847 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.580851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:53.581094 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:53.581143 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 11:33:53.581220 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 11:33:53.581300 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":112}
I20260926 11:33:53.581503 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:53.577252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54823 (local address 127.0.98.3:42313)
0926 11:33:53.577398 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:53.577402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:53.577545 (+ 143us) server_negotiation.cc:407] Connection header received
0926 11:33:53.577636 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:53.577638 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:53.577716 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:53.577787 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:53.578007 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.578701 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.580115 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.580719 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.580821 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:53.580884 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:53.581287 (+ 403us) server_negotiation.cc:299] Negotiation successful
0926 11:33:53.581368 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 11:33:53.581429 20075 raft_consensus.cc:504] T 67dc72125e894703aa54ebf373f71370 P 77fe14a61a29437e93296b08f8a8cff4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe14a61a29437e93296b08f8a8cff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40783 } } peers { permanent_uuid: "b5e4dead5cd145b3a2ce627c1bb4df0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42313 } } peers { permanent_uuid: "1a432b3c1a5e4c4f9dd195a2b4efcfbc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42151 } }
I20260926 11:33:53.581956 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:53.576930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39629 (local address 127.0.98.1:40783)
0926 11:33:53.577085 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:53.577089 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:53.577106 (+ 17us) server_negotiation.cc:407] Connection header received
0926 11:33:53.577236 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:53.577247 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:53.577310 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:53.577390 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:53.577730 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.578529 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.580115 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.580705 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.580821 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:53.580906 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:53.581763 (+ 857us) server_negotiation.cc:299] Negotiation successful
0926 11:33:53.581840 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 11:33:53.582065 20076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:53.577105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42313 (local address 127.0.98.2:54823)
0926 11:33:53.577453 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:53.577467 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:53.577558 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:53.577806 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:53.577808 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:53.577823 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:53.577904 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:53.577908 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.578794 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.578797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:53.579989 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:53.580000 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.580847 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.580851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:53.581100 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:53.581141 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 11:33:53.581254 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 11:33:53.581300 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":102,"thread_start_us":167,"threads_started":1}
I20260926 11:33:53.582505 19736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dc72125e894703aa54ebf373f71370" candidate_uuid: "1a432b3c1a5e4c4f9dd195a2b4efcfbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77fe14a61a29437e93296b08f8a8cff4" is_pre_election: true
I20260926 11:33:53.582631 20004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dc72125e894703aa54ebf373f71370" candidate_uuid: "1a432b3c1a5e4c4f9dd195a2b4efcfbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5e4dead5cd145b3a2ce627c1bb4df0f" is_pre_election: true
I20260926 11:33:53.582706 19736 raft_consensus.cc:2454] T 67dc72125e894703aa54ebf373f71370 P 77fe14a61a29437e93296b08f8a8cff4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a432b3c1a5e4c4f9dd195a2b4efcfbc in term 0.
I20260926 11:33:53.582943 20004 raft_consensus.cc:2454] T 67dc72125e894703aa54ebf373f71370 P b5e4dead5cd145b3a2ce627c1bb4df0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a432b3c1a5e4c4f9dd195a2b4efcfbc in term 0.
I20260926 11:33:53.583349 19803 leader_election.cc:302] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a432b3c1a5e4c4f9dd195a2b4efcfbc, b5e4dead5cd145b3a2ce627c1bb4df0f; no voters:
I20260926 11:33:53.583667 20073 raft_consensus.cc:2781] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:33:53.583925 20073 raft_consensus.cc:482] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:33:53.583998 20073 raft_consensus.cc:3037] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:33:53.584013 20075 leader_election.cc:288] T 67dc72125e894703aa54ebf373f71370 P 77fe14a61a29437e93296b08f8a8cff4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5e4dead5cd145b3a2ce627c1bb4df0f (127.0.98.3:42313), 1a432b3c1a5e4c4f9dd195a2b4efcfbc (127.0.98.2:42151)
I20260926 11:33:53.585265 20073 raft_consensus.cc:504] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe14a61a29437e93296b08f8a8cff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40783 } } peers { permanent_uuid: "b5e4dead5cd145b3a2ce627c1bb4df0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42313 } } peers { permanent_uuid: "1a432b3c1a5e4c4f9dd195a2b4efcfbc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42151 } }
I20260926 11:33:53.585635 20073 leader_election.cc:288] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc [CANDIDATE]: Term 1 election: Requested vote from peers 77fe14a61a29437e93296b08f8a8cff4 (127.0.98.1:40783), b5e4dead5cd145b3a2ce627c1bb4df0f (127.0.98.3:42313)
I20260926 11:33:53.586006 19736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dc72125e894703aa54ebf373f71370" candidate_uuid: "1a432b3c1a5e4c4f9dd195a2b4efcfbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77fe14a61a29437e93296b08f8a8cff4"
I20260926 11:33:53.586138 20004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dc72125e894703aa54ebf373f71370" candidate_uuid: "1a432b3c1a5e4c4f9dd195a2b4efcfbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5e4dead5cd145b3a2ce627c1bb4df0f"
I20260926 11:33:53.586146 19736 raft_consensus.cc:3037] T 67dc72125e894703aa54ebf373f71370 P 77fe14a61a29437e93296b08f8a8cff4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:33:53.586246 20004 raft_consensus.cc:3037] T 67dc72125e894703aa54ebf373f71370 P b5e4dead5cd145b3a2ce627c1bb4df0f [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:33:53.587487 19736 raft_consensus.cc:2454] T 67dc72125e894703aa54ebf373f71370 P 77fe14a61a29437e93296b08f8a8cff4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a432b3c1a5e4c4f9dd195a2b4efcfbc in term 1.
I20260926 11:33:53.587489 20004 raft_consensus.cc:2454] T 67dc72125e894703aa54ebf373f71370 P b5e4dead5cd145b3a2ce627c1bb4df0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a432b3c1a5e4c4f9dd195a2b4efcfbc in term 1.
I20260926 11:33:53.587752 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:53.583012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42313 (local address 127.0.98.1:33841)
0926 11:33:53.583307 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:53.583320 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:53.583420 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:53.583719 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:53.583722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:53.583764 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:53.583870 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:53.583877 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.584808 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.584811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:53.586095 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:53.586106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.587050 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.587053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:53.587146 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:53.587169 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 11:33:53.587511 (+ 342us) client_negotiation.cc:240] Negotiation successful
0926 11:33:53.587584 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":96,"threads_started":1}
I20260926 11:33:53.587471 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:53.583080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33841 (local address 127.0.98.3:42313)
0926 11:33:53.583180 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:53.583187 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:53.583400 (+ 213us) server_negotiation.cc:407] Connection header received
0926 11:33:53.583502 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:53.583505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:53.583548 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:53.583620 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:53.584010 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.584670 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.586233 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.586827 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.586901 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:53.586958 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:53.587279 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 11:33:53.587346 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 11:33:53.588085 19803 leader_election.cc:302] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a432b3c1a5e4c4f9dd195a2b4efcfbc, b5e4dead5cd145b3a2ce627c1bb4df0f; no voters:
I20260926 11:33:53.588186 20004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dc72125e894703aa54ebf373f71370" candidate_uuid: "77fe14a61a29437e93296b08f8a8cff4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5e4dead5cd145b3a2ce627c1bb4df0f" is_pre_election: true
I20260926 11:33:53.588342 20065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:53.583952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53053 (local address 127.0.98.2:42151)
0926 11:33:53.584049 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:53.584054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:53.584348 (+ 294us) server_negotiation.cc:407] Connection header received
0926 11:33:53.584438 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:53.584441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:53.584481 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:53.584551 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:53.584813 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.585553 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.587063 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.587616 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.587687 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:53.587772 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:53.588131 (+ 359us) server_negotiation.cc:299] Negotiation successful
0926 11:33:53.588202 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 11:33:53.588356 20004 raft_consensus.cc:2379] T 67dc72125e894703aa54ebf373f71370 P b5e4dead5cd145b3a2ce627c1bb4df0f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77fe14a61a29437e93296b08f8a8cff4 in current term 1: Already voted for candidate 1a432b3c1a5e4c4f9dd195a2b4efcfbc in this term.
I20260926 11:33:53.588227 20073 raft_consensus.cc:2781] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:33:53.588519 20078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:53.583549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42151 (local address 127.0.98.1:53053)
0926 11:33:53.584251 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:53.584266 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:53.584363 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:53.584589 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:53.584591 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:53.584607 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:53.584708 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:53.584713 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.585680 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.585683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:53.586933 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:53.586942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.587874 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.587878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:53.587988 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:53.588014 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 11:33:53.588254 (+ 240us) client_negotiation.cc:240] Negotiation successful
0926 11:33:53.588322 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":146,"threads_started":1}
I20260926 11:33:53.588981 20073 raft_consensus.cc:686] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc [term 1 LEADER]: Becoming Leader. State: Replica: 1a432b3c1a5e4c4f9dd195a2b4efcfbc, State: Running, Role: LEADER
I20260926 11:33:53.589114 19870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67dc72125e894703aa54ebf373f71370" candidate_uuid: "77fe14a61a29437e93296b08f8a8cff4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a432b3c1a5e4c4f9dd195a2b4efcfbc" is_pre_election: true
I20260926 11:33:53.589334 20073 consensus_queue.cc:237] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe14a61a29437e93296b08f8a8cff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40783 } } peers { permanent_uuid: "b5e4dead5cd145b3a2ce627c1bb4df0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42313 } } peers { permanent_uuid: "1a432b3c1a5e4c4f9dd195a2b4efcfbc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42151 } }
I20260926 11:33:53.589704 19669 leader_election.cc:302] T 67dc72125e894703aa54ebf373f71370 P 77fe14a61a29437e93296b08f8a8cff4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77fe14a61a29437e93296b08f8a8cff4; no voters: 1a432b3c1a5e4c4f9dd195a2b4efcfbc, b5e4dead5cd145b3a2ce627c1bb4df0f
I20260926 11:33:53.589962 20075 raft_consensus.cc:2726] T 67dc72125e894703aa54ebf373f71370 P 77fe14a61a29437e93296b08f8a8cff4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 11:33:53.592675 19591 catalog_manager.cc:5495] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a432b3c1a5e4c4f9dd195a2b4efcfbc (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1a432b3c1a5e4c4f9dd195a2b4efcfbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77fe14a61a29437e93296b08f8a8cff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5e4dead5cd145b3a2ce627c1bb4df0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a432b3c1a5e4c4f9dd195a2b4efcfbc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42151 } health_report { overall_health: HEALTHY } } }
I20260926 11:33:53.604630 20065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:53.601077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38248 (local address 127.0.98.2:42151)
0926 11:33:53.601216 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:53.601220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:53.601234 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:33:53.601339 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:53.601341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:53.601407 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:53.601512 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:33:53.601767 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.602266 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.603124 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.603345 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.603460 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:53.603814 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:33:53.604222 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:33:53.604350 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:53.604414 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 11:33:53.604467 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 11:33:53.610363 20004 raft_consensus.cc:1261] T 67dc72125e894703aa54ebf373f71370 P b5e4dead5cd145b3a2ce627c1bb4df0f [term 1 FOLLOWER]: Refusing update from remote peer 1a432b3c1a5e4c4f9dd195a2b4efcfbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:33:53.610428 19736 raft_consensus.cc:1261] T 67dc72125e894703aa54ebf373f71370 P 77fe14a61a29437e93296b08f8a8cff4 [term 1 FOLLOWER]: Refusing update from remote peer 1a432b3c1a5e4c4f9dd195a2b4efcfbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:33:53.611044 20073 consensus_queue.cc:1035] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5e4dead5cd145b3a2ce627c1bb4df0f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:33:53.611250 20079 consensus_queue.cc:1035] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "77fe14a61a29437e93296b08f8a8cff4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:33:53.619154 20082 mvcc.cc:205] Tried to move back new op lower bound from 7333570288061505536 to 7333570287985090560. Current Snapshot: MvccSnapshot[applied={T|T < 7333570288061505536}]
I20260926 11:33:53.620746 20083 mvcc.cc:205] Tried to move back new op lower bound from 7333570288061505536 to 7333570287985090560. Current Snapshot: MvccSnapshot[applied={T|T < 7333570288061505536}]
I20260926 11:33:53.627251 19591 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:33:53.627444 19591 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:33:53.629833 19591 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 379f0f478d61496eac7380f7c947e2ef: Sending DeleteTablet for 3 replicas of tablet 67dc72125e894703aa54ebf373f71370
I20260926 11:33:53.630534 19716 tablet_service.cc:1514] Processing DeleteTablet for tablet 67dc72125e894703aa54ebf373f71370 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:33:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60014
I20260926 11:33:53.630659 19850 tablet_service.cc:1514] Processing DeleteTablet for tablet 67dc72125e894703aa54ebf373f71370 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:33:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38234
I20260926 11:33:53.630739 19984 tablet_service.cc:1514] Processing DeleteTablet for tablet 67dc72125e894703aa54ebf373f71370 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:33:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39414
I20260926 11:33:53.631107 20092 tablet_replica.cc:330] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc: stopping tablet replica
I20260926 11:33:53.631138 20091 tablet_replica.cc:330] T 67dc72125e894703aa54ebf373f71370 P 77fe14a61a29437e93296b08f8a8cff4: stopping tablet replica
I20260926 11:33:53.631433 20092 raft_consensus.cc:2229] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:33:53.631440 20091 raft_consensus.cc:2229] T 67dc72125e894703aa54ebf373f71370 P 77fe14a61a29437e93296b08f8a8cff4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:33:53.631994 20093 tablet_replica.cc:330] T 67dc72125e894703aa54ebf373f71370 P b5e4dead5cd145b3a2ce627c1bb4df0f: stopping tablet replica
I20260926 11:33:53.632308 20093 raft_consensus.cc:2229] T 67dc72125e894703aa54ebf373f71370 P b5e4dead5cd145b3a2ce627c1bb4df0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:33:53.632484 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 19653
I20260926 11:33:53.633402 20093 raft_consensus.cc:2258] T 67dc72125e894703aa54ebf373f71370 P b5e4dead5cd145b3a2ce627c1bb4df0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:33:53.633553 20092 raft_consensus.cc:2258] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:33:53.634927 20093 ts_tablet_manager.cc:1905] T 67dc72125e894703aa54ebf373f71370 P b5e4dead5cd145b3a2ce627c1bb4df0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:33:53.634982 20092 ts_tablet_manager.cc:1905] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:33:53.637701 20093 ts_tablet_manager.cc:1918] T 67dc72125e894703aa54ebf373f71370 P b5e4dead5cd145b3a2ce627c1bb4df0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:33:53.637837 20093 log.cc:1196] T 67dc72125e894703aa54ebf373f71370 P b5e4dead5cd145b3a2ce627c1bb4df0f: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-2/wal/wals/67dc72125e894703aa54ebf373f71370
I20260926 11:33:53.638198 20093 ts_tablet_manager.cc:1939] T 67dc72125e894703aa54ebf373f71370 P b5e4dead5cd145b3a2ce627c1bb4df0f: Deleting consensus metadata
I20260926 11:33:53.638340 20092 ts_tablet_manager.cc:1918] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:33:53.638474 20092 log.cc:1196] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790422180210734-392-0/minicluster-data/ts-1/wal/wals/67dc72125e894703aa54ebf373f71370
I20260926 11:33:53.638836 20092 ts_tablet_manager.cc:1939] T 67dc72125e894703aa54ebf373f71370 P 1a432b3c1a5e4c4f9dd195a2b4efcfbc: Deleting consensus metadata
I20260926 11:33:53.639521 19575 catalog_manager.cc:4841] TS b5e4dead5cd145b3a2ce627c1bb4df0f (127.0.98.3:42313): tablet 67dc72125e894703aa54ebf373f71370 (table test-table [id=1eab97f6dc114e05b6edeed988bb1c65]) successfully deleted
I20260926 11:33:53.640103 19575 catalog_manager.cc:4841] TS 1a432b3c1a5e4c4f9dd195a2b4efcfbc (127.0.98.2:42151): tablet 67dc72125e894703aa54ebf373f71370 (table test-table [id=1eab97f6dc114e05b6edeed988bb1c65]) successfully deleted
W20260926 11:33:53.641110 19575 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40783 (error 108)
I20260926 11:33:53.641338 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 19787
I20260926 11:33:53.642777 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:53.642405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40783 (local address 127.0.0.1:60022)
0926 11:33:53.642598 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:53.642672 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260926 11:33:53.642977 19575 catalog_manager.cc:4584] TS 77fe14a61a29437e93296b08f8a8cff4 (127.0.98.1:40783): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 67dc72125e894703aa54ebf373f71370: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40783: connect: Connection refused (error 111)
I20260926 11:33:53.647295 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 19921
I20260926 11:33:53.652688 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 19559
2026-09-26T11:33:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2251 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T11:33:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:33:53Z Disabled control of system clock
I20260926 11:33:53.682873 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44477
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 11:33:53.794101 20098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:53.794370 20098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:53.794407 20098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:53.798417 20098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:33:53.798470 20098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:53.798486 20098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:53.798503 20098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:33:53.798519 20098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:33:53.803208 20098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44477
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:53.804243 20098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:53.805495 20098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:33:53.813819 20106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:53.813830 20103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:53.813802 20104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:53.814134 20098 server_base.cc:1034] running on GCE node
I20260926 11:33:53.814821 20098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:53.816061 20098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:53.817245 20098 hybrid_clock.cc:648] HybridClock initialized: now 1790422433817215 us; error 40 us; skew 500 ppm
I20260926 11:33:53.819099 20098 webserver.cc:460] Webserver started at http://127.0.98.62:36375/ using document root <none> and password file <none>
I20260926 11:33:53.819681 20098 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:53.819803 20098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:53.820029 20098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:33:53.822062 20098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "46e6dfb706834743a3568306f96183a7"
format_stamp: "Formatted at 2026-09-26 11:33:53 on dist-test-slave-mrck"
server_key: "bf3e9519eff8fff8c558514e8b22673a"
server_key_iv: "1c7406f67e02db153eb5ee98a96f6743"
server_key_version: "encryptionkey@0"
I20260926 11:33:53.822633 20098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "46e6dfb706834743a3568306f96183a7"
format_stamp: "Formatted at 2026-09-26 11:33:53 on dist-test-slave-mrck"
server_key: "bf3e9519eff8fff8c558514e8b22673a"
server_key_iv: "1c7406f67e02db153eb5ee98a96f6743"
server_key_version: "encryptionkey@0"
I20260926 11:33:53.826265 20098 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.002s
I20260926 11:33:53.828640 20112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:53.829787 20098 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:33:53.829922 20098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "46e6dfb706834743a3568306f96183a7"
format_stamp: "Formatted at 2026-09-26 11:33:53 on dist-test-slave-mrck"
server_key: "bf3e9519eff8fff8c558514e8b22673a"
server_key_iv: "1c7406f67e02db153eb5ee98a96f6743"
server_key_version: "encryptionkey@0"
I20260926 11:33:53.830026 20098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:53.846290 20098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:53.847031 20098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:53.847226 20098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:53.854010 20098 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44477
I20260926 11:33:53.854018 20164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44477 every 8 connection(s)
I20260926 11:33:53.854954 20098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:33:53.857960 20165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:33:53.858419 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 20098
I20260926 11:33:53.858566 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:33:53.858855 392 external_mini_cluster.cc:1428] Setting key 9514bf33c5d2d5d2ef727b64a1084d10
I20260926 11:33:53.864176 20165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46e6dfb706834743a3568306f96183a7: Bootstrap starting.
I20260926 11:33:53.866768 20165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46e6dfb706834743a3568306f96183a7: Neither blocks nor log segments found. Creating new log.
I20260926 11:33:53.867575 20165 log.cc:824] T 00000000000000000000000000000000 P 46e6dfb706834743a3568306f96183a7: Log is configured to *not* fsync() on all Append() calls
I20260926 11:33:53.868142 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:53.860220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45508 (local address 127.0.98.62:44477)
0926 11:33:53.860798 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:53.860814 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:53.860849 (+ 35us) server_negotiation.cc:407] Connection header received
0926 11:33:53.861487 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:53.861503 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:53.861898 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:53.862178 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:53.862566 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.863508 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.864683 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:53.865141 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:53.865265 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:53.866150 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:53.866155 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:53.866167 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:33:53.866580 (+ 413us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:53.866637 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:33:53.866657 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:53.866733 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:53.866975 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 11:33:53.867268 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":127,"threads_started":1}
I20260926 11:33:53.869760 20165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46e6dfb706834743a3568306f96183a7: No bootstrap required, opened a new log
I20260926 11:33:53.873320 20165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46e6dfb706834743a3568306f96183a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e6dfb706834743a3568306f96183a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44477 } }
I20260926 11:33:53.873548 20165 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46e6dfb706834743a3568306f96183a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:33:53.873641 20165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46e6dfb706834743a3568306f96183a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46e6dfb706834743a3568306f96183a7, State: Initialized, Role: FOLLOWER
I20260926 11:33:53.874027 20165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46e6dfb706834743a3568306f96183a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e6dfb706834743a3568306f96183a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44477 } }
I20260926 11:33:53.874136 20165 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46e6dfb706834743a3568306f96183a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:33:53.874174 20165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46e6dfb706834743a3568306f96183a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:33:53.874241 20165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46e6dfb706834743a3568306f96183a7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:33:53.875188 20165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46e6dfb706834743a3568306f96183a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e6dfb706834743a3568306f96183a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44477 } }
I20260926 11:33:53.875481 20165 leader_election.cc:302] T 00000000000000000000000000000000 P 46e6dfb706834743a3568306f96183a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46e6dfb706834743a3568306f96183a7; no voters:
I20260926 11:33:53.875772 20165 leader_election.cc:288] T 00000000000000000000000000000000 P 46e6dfb706834743a3568306f96183a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:33:53.876024 20170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46e6dfb706834743a3568306f96183a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:33:53.876276 20170 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46e6dfb706834743a3568306f96183a7 [term 1 LEADER]: Becoming Leader. State: Replica: 46e6dfb706834743a3568306f96183a7, State: Running, Role: LEADER
I20260926 11:33:53.876674 20170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46e6dfb706834743a3568306f96183a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e6dfb706834743a3568306f96183a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44477 } }
I20260926 11:33:53.877051 20165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46e6dfb706834743a3568306f96183a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:33:53.878319 20172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46e6dfb706834743a3568306f96183a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46e6dfb706834743a3568306f96183a7. Latest consensus state: current_term: 1 leader_uuid: "46e6dfb706834743a3568306f96183a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e6dfb706834743a3568306f96183a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44477 } } }
I20260926 11:33:53.878371 20171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46e6dfb706834743a3568306f96183a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46e6dfb706834743a3568306f96183a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e6dfb706834743a3568306f96183a7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44477 } } }
I20260926 11:33:53.878456 20172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46e6dfb706834743a3568306f96183a7 [sys.catalog]: This master's current role is: LEADER
I20260926 11:33:53.878479 20171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46e6dfb706834743a3568306f96183a7 [sys.catalog]: This master's current role is: LEADER
I20260926 11:33:53.878844 20179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:33:53.881609 20179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:33:53.887279 20179 catalog_manager.cc:1347] Generated new cluster ID: 3c3f92192abc495286b3d90a91657996
I20260926 11:33:53.887367 20179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:33:53.904009 20179 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:33:53.905326 20179 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:33:53.915936 20179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46e6dfb706834743a3568306f96183a7: Generated new TSK 0
I20260926 11:33:53.916750 20179 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:33:53.926203 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44477
--builtin_ntp_servers=127.0.98.20:45809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 11:33:54.046691 20189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:54.046944 20189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:54.046981 20189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:54.050844 20189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:54.050895 20189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:54.050948 20189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 11:33:54.055455 20189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:54.056522 20189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:54.057854 20189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:33:54.067226 20194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:54.067312 20195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:54.067226 20197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:54.067584 20189 server_base.cc:1034] running on GCE node
I20260926 11:33:54.068073 20189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:54.068667 20189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:54.069883 20189 hybrid_clock.cc:648] HybridClock initialized: now 1790422434069860 us; error 43 us; skew 500 ppm
I20260926 11:33:54.071702 20189 webserver.cc:460] Webserver started at http://127.0.98.1:37977/ using document root <none> and password file <none>
I20260926 11:33:54.072295 20189 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:54.072356 20189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:54.072530 20189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:33:54.074467 20189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-0/data/instance:
uuid: "9cd11d467628470984d8ebd33bdafd6c"
format_stamp: "Formatted at 2026-09-26 11:33:54 on dist-test-slave-mrck"
server_key: "3a0b6b1ec6c0764af7f27205dc7cfd4e"
server_key_iv: "ad89cec6788ce97f7018c1f4ad30622c"
server_key_version: "encryptionkey@0"
I20260926 11:33:54.074946 20189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9cd11d467628470984d8ebd33bdafd6c"
format_stamp: "Formatted at 2026-09-26 11:33:54 on dist-test-slave-mrck"
server_key: "3a0b6b1ec6c0764af7f27205dc7cfd4e"
server_key_iv: "ad89cec6788ce97f7018c1f4ad30622c"
server_key_version: "encryptionkey@0"
I20260926 11:33:54.078720 20189 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 11:33:54.081185 20203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:54.082372 20189 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 11:33:54.082480 20189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-0/wal
uuid: "9cd11d467628470984d8ebd33bdafd6c"
format_stamp: "Formatted at 2026-09-26 11:33:54 on dist-test-slave-mrck"
server_key: "3a0b6b1ec6c0764af7f27205dc7cfd4e"
server_key_iv: "ad89cec6788ce97f7018c1f4ad30622c"
server_key_version: "encryptionkey@0"
I20260926 11:33:54.082567 20189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:54.093550 20189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:54.094172 20189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:54.094316 20189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:54.094834 20189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:33:54.095703 20189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:33:54.095819 20189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:54.095863 20189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:33:54.095884 20189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:54.104944 20189 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46629
I20260926 11:33:54.104974 20316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46629 every 8 connection(s)
I20260926 11:33:54.105880 20189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-0/data/info.pb
I20260926 11:33:54.112999 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 20189
I20260926 11:33:54.113150 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-0/wal/instance
I20260926 11:33:54.113456 392 external_mini_cluster.cc:1428] Setting key 10214134ecea5c60ddd8582ff656d764
I20260926 11:33:54.115357 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:54.107724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45355 (local address 127.0.98.62:44477)
0926 11:33:54.107892 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:54.107896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:54.108943 (+ 1047us) server_negotiation.cc:407] Connection header received
0926 11:33:54.110323 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:54.110326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:54.110384 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:54.110465 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:54.111344 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.111966 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.113116 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.113406 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.113520 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:54.114784 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:54.114789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:54.114797 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:33:54.114813 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:54.114841 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:33:54.114852 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:54.114931 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:54.115150 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 11:33:54.115216 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 11:33:54.115832 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44477
--builtin_ntp_servers=127.0.98.20:45809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 11:33:54.116387 20319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:54.108163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44477 (local address 127.0.98.1:45355)
0926 11:33:54.108789 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:54.108828 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:54.110111 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:54.110633 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:54.110640 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:54.110723 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:33:54.111203 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:54.111215 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.112091 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.112095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:54.112990 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:54.112999 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.113560 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.113564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:54.113728 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:54.114438 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:33:54.114441 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:33:54.114493 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:33:54.114572 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:33:54.114950 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:33:54.114960 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:33:54.114967 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 11:33:54.115108 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 11:33:54.115362 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":105,"threads_started":1}
I20260926 11:33:54.117645 20317 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44477
I20260926 11:33:54.117924 20317 heartbeater.cc:464] Registering TS with master...
I20260926 11:33:54.118533 20317 heartbeater.cc:511] Master 127.0.98.62:44477 requested a full tablet report, sending...
I20260926 11:33:54.120074 20129 ts_manager.cc:194] Registered new tserver with Master: 9cd11d467628470984d8ebd33bdafd6c (127.0.98.1:46629)
I20260926 11:33:54.121228 20129 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:45355
W20260926 11:33:54.229442 20320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:54.229728 20320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:54.229769 20320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:54.234035 20320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:54.234092 20320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:54.234197 20320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 11:33:54.239044 20320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:54.240278 20320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:54.241664 20320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:33:54.251129 20326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:54.251324 20320 server_base.cc:1034] running on GCE node
W20260926 11:33:54.251148 20325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:54.251180 20328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:54.251912 20320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:54.252620 20320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:54.253779 20320 hybrid_clock.cc:648] HybridClock initialized: now 1790422434253743 us; error 49 us; skew 500 ppm
I20260926 11:33:54.255713 20320 webserver.cc:460] Webserver started at http://127.0.98.2:44189/ using document root <none> and password file <none>
I20260926 11:33:54.256374 20320 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:54.256471 20320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:54.256709 20320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:33:54.258742 20320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-1/data/instance:
uuid: "aae445ca50a540609a68b80c428da496"
format_stamp: "Formatted at 2026-09-26 11:33:54 on dist-test-slave-mrck"
server_key: "5a621bf81839c827f27490c903b3900c"
server_key_iv: "7a31a2d185674c9448354bba0242b978"
server_key_version: "encryptionkey@0"
I20260926 11:33:54.259310 20320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-1/wal/instance:
uuid: "aae445ca50a540609a68b80c428da496"
format_stamp: "Formatted at 2026-09-26 11:33:54 on dist-test-slave-mrck"
server_key: "5a621bf81839c827f27490c903b3900c"
server_key_iv: "7a31a2d185674c9448354bba0242b978"
server_key_version: "encryptionkey@0"
I20260926 11:33:54.263016 20320 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260926 11:33:54.265450 20334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:54.266652 20320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 11:33:54.266790 20320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-1/wal
uuid: "aae445ca50a540609a68b80c428da496"
format_stamp: "Formatted at 2026-09-26 11:33:54 on dist-test-slave-mrck"
server_key: "5a621bf81839c827f27490c903b3900c"
server_key_iv: "7a31a2d185674c9448354bba0242b978"
server_key_version: "encryptionkey@0"
I20260926 11:33:54.266899 20320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:54.277240 20320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:54.277998 20320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:54.278187 20320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:54.278800 20320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:33:54.279760 20320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:33:54.279834 20320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:54.279896 20320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:33:54.279927 20320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:54.288594 20320 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40463
I20260926 11:33:54.288614 20447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40463 every 8 connection(s)
I20260926 11:33:54.289491 20320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-1/data/info.pb
I20260926 11:33:54.293243 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 20320
I20260926 11:33:54.293350 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-1/wal/instance
I20260926 11:33:54.293649 392 external_mini_cluster.cc:1428] Setting key 704831d23213e20dd85ebae32999ba26
I20260926 11:33:54.295915 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44477
--builtin_ntp_servers=127.0.98.20:45809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 11:33:54.300213 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:54.291381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49251 (local address 127.0.98.62:44477)
0926 11:33:54.291530 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:54.291534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:54.292615 (+ 1081us) server_negotiation.cc:407] Connection header received
0926 11:33:54.294215 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:54.294220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:54.294276 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:54.294375 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:54.295639 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.296419 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.298025 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.298253 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.298344 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:54.299645 (+ 1301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:54.299648 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:54.299651 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:33:54.299663 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:54.299682 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:33:54.299690 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:54.299833 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:54.300047 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 11:33:54.300102 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 11:33:54.301529 20450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:54.291806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44477 (local address 127.0.98.2:49251)
0926 11:33:54.292457 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:54.292497 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:54.293939 (+ 1442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:54.294583 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:54.294592 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:54.294699 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:33:54.295464 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:54.295480 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.296548 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.296552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:54.297887 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:54.297901 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.298415 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.298422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:54.298669 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:54.299329 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:33:54.299333 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:33:54.299393 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:33:54.299483 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:33:54.299869 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:33:54.299881 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:33:54.299889 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 11:33:54.300033 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 11:33:54.300370 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":113,"threads_started":1}
I20260926 11:33:54.302791 20448 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44477
I20260926 11:33:54.303077 20448 heartbeater.cc:464] Registering TS with master...
I20260926 11:33:54.303815 20448 heartbeater.cc:511] Master 127.0.98.62:44477 requested a full tablet report, sending...
I20260926 11:33:54.305089 20129 ts_manager.cc:194] Registered new tserver with Master: aae445ca50a540609a68b80c428da496 (127.0.98.2:40463)
I20260926 11:33:54.305980 20129 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:49251
W20260926 11:33:54.409755 20451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:54.410010 20451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:54.410049 20451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:54.414021 20451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:54.414068 20451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:54.414120 20451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 11:33:54.419010 20451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:54.420070 20451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:54.421341 20451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:33:54.430650 20456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:54.430655 20457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:54.430686 20459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:54.431000 20451 server_base.cc:1034] running on GCE node
I20260926 11:33:54.431401 20451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:54.432127 20451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:54.433313 20451 hybrid_clock.cc:648] HybridClock initialized: now 1790422434433297 us; error 35 us; skew 500 ppm
I20260926 11:33:54.435258 20451 webserver.cc:460] Webserver started at http://127.0.98.3:38185/ using document root <none> and password file <none>
I20260926 11:33:54.435906 20451 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:54.436002 20451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:54.436228 20451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:33:54.438264 20451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-2/data/instance:
uuid: "fbe5d43e75374517b4462a3bc596977d"
format_stamp: "Formatted at 2026-09-26 11:33:54 on dist-test-slave-mrck"
server_key: "d01423776cdee4cb3ea26e7cdde33364"
server_key_iv: "aa6fd9e73dbaba1d261015e821761a71"
server_key_version: "encryptionkey@0"
I20260926 11:33:54.438843 20451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fbe5d43e75374517b4462a3bc596977d"
format_stamp: "Formatted at 2026-09-26 11:33:54 on dist-test-slave-mrck"
server_key: "d01423776cdee4cb3ea26e7cdde33364"
server_key_iv: "aa6fd9e73dbaba1d261015e821761a71"
server_key_version: "encryptionkey@0"
I20260926 11:33:54.442545 20451 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 11:33:54.445096 20465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:54.446261 20451 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 11:33:54.446412 20451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-2/wal
uuid: "fbe5d43e75374517b4462a3bc596977d"
format_stamp: "Formatted at 2026-09-26 11:33:54 on dist-test-slave-mrck"
server_key: "d01423776cdee4cb3ea26e7cdde33364"
server_key_iv: "aa6fd9e73dbaba1d261015e821761a71"
server_key_version: "encryptionkey@0"
I20260926 11:33:54.446525 20451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:54.460117 20451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:54.460870 20451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:54.461086 20451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:54.461695 20451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 11:33:54.462772 20451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 11:33:54.462860 20451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:54.462935 20451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 11:33:54.462990 20451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:54.472522 20451 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41137
I20260926 11:33:54.472581 20578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41137 every 8 connection(s)
I20260926 11:33:54.473541 20451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-2/data/info.pb
I20260926 11:33:54.482554 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 20451
I20260926 11:33:54.482697 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-2/wal/instance
I20260926 11:33:54.483012 392 external_mini_cluster.cc:1428] Setting key fa3e095d46f4cee114884456f7c9194e
I20260926 11:33:54.483158 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:54.475467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48181 (local address 127.0.98.62:44477)
0926 11:33:54.475611 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:54.475615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:54.476745 (+ 1130us) server_negotiation.cc:407] Connection header received
0926 11:33:54.478336 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:54.478340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:54.478411 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:54.478502 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:54.479463 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.480075 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.481234 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.481462 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.481576 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:54.482632 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:54.482636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:54.482639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:33:54.482659 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:54.482697 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:33:54.482708 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:54.482772 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:54.482946 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 11:33:54.483013 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 11:33:54.484262 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:54.475912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44477 (local address 127.0.98.3:48181)
0926 11:33:54.476595 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:54.476637 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:54.478027 (+ 1390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:54.478690 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:54.478697 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:54.478784 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:33:54.479325 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:54.479340 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.480199 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.480202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:54.481108 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:54.481117 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.481613 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.481617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:54.481790 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:54.482362 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 11:33:54.482363 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:33:54.482417 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 11:33:54.482481 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:33:54.482785 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:33:54.482795 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:33:54.482802 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 11:33:54.482936 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 11:33:54.483241 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":103,"threads_started":1}
I20260926 11:33:54.485343 20579 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44477
I20260926 11:33:54.485610 20579 heartbeater.cc:464] Registering TS with master...
I20260926 11:33:54.486152 20579 heartbeater.cc:511] Master 127.0.98.62:44477 requested a full tablet report, sending...
I20260926 11:33:54.487169 20129 ts_manager.cc:194] Registered new tserver with Master: fbe5d43e75374517b4462a3bc596977d (127.0.98.3:41137)
I20260926 11:33:54.487721 20129 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:48181
I20260926 11:33:54.497095 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 11:33:54.502583 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:54.498997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45510 (local address 127.0.98.62:44477)
0926 11:33:54.499147 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:54.499159 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:54.499310 (+ 151us) server_negotiation.cc:407] Connection header received
0926 11:33:54.499387 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:54.499389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:54.499435 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:54.499515 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:54.499900 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.500410 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.501215 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.501434 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.501525 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:54.502247 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:54.502250 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:54.502252 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:33:54.502264 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:54.502282 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:33:54.502289 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:54.502351 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:54.502433 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 11:33:54.502478 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 11:33:54.505391 20129 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 11:33:54.508443 20129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 11:33:54.522709 20591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:54.516490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46629 (local address 127.0.0.1:40956)
0926 11:33:54.517116 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:54.517171 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:54.517302 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:54.518348 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:54.518353 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:54.518371 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:54.518557 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:54.518564 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.519945 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.519948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:54.521390 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:54.521398 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.522179 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.522183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:54.522285 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:54.522321 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 11:33:54.522428 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 11:33:54.522548 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":145,"thread_start_us":80,"threads_started":1}
I20260926 11:33:54.522713 20592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:54.516623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40956 (local address 127.0.98.1:46629)
0926 11:33:54.517624 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:54.517640 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:54.517665 (+ 25us) server_negotiation.cc:407] Connection header received
0926 11:33:54.517744 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:54.517749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:54.517997 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:54.518160 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:54.518713 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.519827 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.521533 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.522056 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.522180 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:54.522377 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:54.522476 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 11:33:54.522570 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":885,"thread_start_us":108,"threads_started":1}
I20260926 11:33:54.523483 20590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:54.516590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41137 (local address 127.0.0.1:56372)
0926 11:33:54.517371 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:54.517386 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:54.517457 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:54.518443 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:54.518446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:54.518458 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:54.518561 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:54.518565 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.519698 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.519702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:54.521360 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:54.521373 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.522635 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.522639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:54.522816 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:54.523150 (+ 334us) client_negotiation.cc:769] Sending connection context
0926 11:33:54.523225 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 11:33:54.523315 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":423,"thread_start_us":104,"threads_started":1}
I20260926 11:33:54.523531 20594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:54.517589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56372 (local address 127.0.98.3:41137)
0926 11:33:54.517996 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:54.518008 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:54.518032 (+ 24us) server_negotiation.cc:407] Connection header received
0926 11:33:54.518104 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:54.518109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:54.518303 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:54.518461 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:54.518658 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.519575 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.521497 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.522286 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.522390 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:54.522569 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:54.523261 (+ 692us) server_negotiation.cc:299] Negotiation successful
0926 11:33:54.523374 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":210,"threads_started":1}
I20260926 11:33:54.523900 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:54.516834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40463 (local address 127.0.0.1:52918)
0926 11:33:54.518126 (+ 1292us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:54.518138 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:54.518209 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:54.518818 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:54.518821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:54.518837 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:54.518976 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:54.518983 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.520519 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.520522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:54.521403 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:54.521414 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.522635 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.522639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:54.522852 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:54.522871 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 11:33:54.522941 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 11:33:54.523007 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1223,"mutex_wait_us":106,"thread_start_us":1131,"threads_started":1}
I20260926 11:33:54.523953 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:54.517188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52918 (local address 127.0.98.2:40463)
0926 11:33:54.518277 (+ 1089us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:54.518289 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:54.518310 (+ 21us) server_negotiation.cc:407] Connection header received
0926 11:33:54.518378 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:54.518384 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:54.518539 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:54.518677 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:54.519116 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.520392 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.521563 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.522286 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.522390 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:54.522569 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:54.523655 (+ 1086us) server_negotiation.cc:299] Negotiation successful
0926 11:33:54.523799 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":994,"thread_start_us":126,"threads_started":1}
I20260926 11:33:54.525720 20251 tablet_service.cc:1467] Processing CreateTablet for tablet 023f28c4dbf541f6a426ecefa09af94a (DEFAULT_TABLE table=test-table [id=dea76d748e554eccabcdcdeddef6db46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:33:54.525871 20513 tablet_service.cc:1467] Processing CreateTablet for tablet 023f28c4dbf541f6a426ecefa09af94a (DEFAULT_TABLE table=test-table [id=dea76d748e554eccabcdcdeddef6db46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:33:54.526599 20251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 023f28c4dbf541f6a426ecefa09af94a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:33:54.526643 20513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 023f28c4dbf541f6a426ecefa09af94a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:33:54.529350 20382 tablet_service.cc:1467] Processing CreateTablet for tablet 023f28c4dbf541f6a426ecefa09af94a (DEFAULT_TABLE table=test-table [id=dea76d748e554eccabcdcdeddef6db46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 11:33:54.530164 20382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 023f28c4dbf541f6a426ecefa09af94a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:33:54.533015 20596 tablet_bootstrap.cc:492] T 023f28c4dbf541f6a426ecefa09af94a P fbe5d43e75374517b4462a3bc596977d: Bootstrap starting.
I20260926 11:33:54.535111 20596 tablet_bootstrap.cc:654] T 023f28c4dbf541f6a426ecefa09af94a P fbe5d43e75374517b4462a3bc596977d: Neither blocks nor log segments found. Creating new log.
I20260926 11:33:54.535880 20596 log.cc:824] T 023f28c4dbf541f6a426ecefa09af94a P fbe5d43e75374517b4462a3bc596977d: Log is configured to *not* fsync() on all Append() calls
I20260926 11:33:54.536640 20598 tablet_bootstrap.cc:492] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496: Bootstrap starting.
I20260926 11:33:54.537587 20599 tablet_bootstrap.cc:492] T 023f28c4dbf541f6a426ecefa09af94a P 9cd11d467628470984d8ebd33bdafd6c: Bootstrap starting.
I20260926 11:33:54.537931 20596 tablet_bootstrap.cc:492] T 023f28c4dbf541f6a426ecefa09af94a P fbe5d43e75374517b4462a3bc596977d: No bootstrap required, opened a new log
I20260926 11:33:54.538107 20596 ts_tablet_manager.cc:1397] T 023f28c4dbf541f6a426ecefa09af94a P fbe5d43e75374517b4462a3bc596977d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260926 11:33:54.538856 20598 tablet_bootstrap.cc:654] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496: Neither blocks nor log segments found. Creating new log.
I20260926 11:33:54.539613 20599 tablet_bootstrap.cc:654] T 023f28c4dbf541f6a426ecefa09af94a P 9cd11d467628470984d8ebd33bdafd6c: Neither blocks nor log segments found. Creating new log.
I20260926 11:33:54.539670 20598 log.cc:824] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496: Log is configured to *not* fsync() on all Append() calls
I20260926 11:33:54.540416 20599 log.cc:824] T 023f28c4dbf541f6a426ecefa09af94a P 9cd11d467628470984d8ebd33bdafd6c: Log is configured to *not* fsync() on all Append() calls
I20260926 11:33:54.541647 20598 tablet_bootstrap.cc:492] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496: No bootstrap required, opened a new log
I20260926 11:33:54.541834 20598 ts_tablet_manager.cc:1397] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 11:33:54.542047 20599 tablet_bootstrap.cc:492] T 023f28c4dbf541f6a426ecefa09af94a P 9cd11d467628470984d8ebd33bdafd6c: No bootstrap required, opened a new log
I20260926 11:33:54.542222 20599 ts_tablet_manager.cc:1397] T 023f28c4dbf541f6a426ecefa09af94a P 9cd11d467628470984d8ebd33bdafd6c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 11:33:54.542420 20596 raft_consensus.cc:348] T 023f28c4dbf541f6a426ecefa09af94a P fbe5d43e75374517b4462a3bc596977d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd11d467628470984d8ebd33bdafd6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46629 } } peers { permanent_uuid: "fbe5d43e75374517b4462a3bc596977d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41137 } } peers { permanent_uuid: "aae445ca50a540609a68b80c428da496" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40463 } }
I20260926 11:33:54.542682 20596 raft_consensus.cc:374] T 023f28c4dbf541f6a426ecefa09af94a P fbe5d43e75374517b4462a3bc596977d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:33:54.542774 20596 raft_consensus.cc:729] T 023f28c4dbf541f6a426ecefa09af94a P fbe5d43e75374517b4462a3bc596977d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbe5d43e75374517b4462a3bc596977d, State: Initialized, Role: FOLLOWER
I20260926 11:33:54.543237 20596 consensus_queue.cc:260] T 023f28c4dbf541f6a426ecefa09af94a P fbe5d43e75374517b4462a3bc596977d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd11d467628470984d8ebd33bdafd6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46629 } } peers { permanent_uuid: "fbe5d43e75374517b4462a3bc596977d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41137 } } peers { permanent_uuid: "aae445ca50a540609a68b80c428da496" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40463 } }
I20260926 11:33:54.543879 20579 heartbeater.cc:503] Master 127.0.98.62:44477 was elected leader, sending a full tablet report...
I20260926 11:33:54.544245 20596 ts_tablet_manager.cc:1428] T 023f28c4dbf541f6a426ecefa09af94a P fbe5d43e75374517b4462a3bc596977d: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260926 11:33:54.545675 20598 raft_consensus.cc:348] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd11d467628470984d8ebd33bdafd6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46629 } } peers { permanent_uuid: "fbe5d43e75374517b4462a3bc596977d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41137 } } peers { permanent_uuid: "aae445ca50a540609a68b80c428da496" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40463 } }
I20260926 11:33:54.545894 20598 raft_consensus.cc:374] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:33:54.545975 20598 raft_consensus.cc:729] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae445ca50a540609a68b80c428da496, State: Initialized, Role: FOLLOWER
I20260926 11:33:54.546428 20598 consensus_queue.cc:260] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd11d467628470984d8ebd33bdafd6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46629 } } peers { permanent_uuid: "fbe5d43e75374517b4462a3bc596977d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41137 } } peers { permanent_uuid: "aae445ca50a540609a68b80c428da496" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40463 } }
I20260926 11:33:54.547065 20448 heartbeater.cc:503] Master 127.0.98.62:44477 was elected leader, sending a full tablet report...
I20260926 11:33:54.547372 20598 ts_tablet_manager.cc:1428] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 11:33:54.548095 20599 raft_consensus.cc:348] T 023f28c4dbf541f6a426ecefa09af94a P 9cd11d467628470984d8ebd33bdafd6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd11d467628470984d8ebd33bdafd6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46629 } } peers { permanent_uuid: "fbe5d43e75374517b4462a3bc596977d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41137 } } peers { permanent_uuid: "aae445ca50a540609a68b80c428da496" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40463 } }
I20260926 11:33:54.548380 20599 raft_consensus.cc:374] T 023f28c4dbf541f6a426ecefa09af94a P 9cd11d467628470984d8ebd33bdafd6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:33:54.548460 20599 raft_consensus.cc:729] T 023f28c4dbf541f6a426ecefa09af94a P 9cd11d467628470984d8ebd33bdafd6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cd11d467628470984d8ebd33bdafd6c, State: Initialized, Role: FOLLOWER
I20260926 11:33:54.548854 20599 consensus_queue.cc:260] T 023f28c4dbf541f6a426ecefa09af94a P 9cd11d467628470984d8ebd33bdafd6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd11d467628470984d8ebd33bdafd6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46629 } } peers { permanent_uuid: "fbe5d43e75374517b4462a3bc596977d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41137 } } peers { permanent_uuid: "aae445ca50a540609a68b80c428da496" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40463 } }
I20260926 11:33:54.549618 20317 heartbeater.cc:503] Master 127.0.98.62:44477 was elected leader, sending a full tablet report...
I20260926 11:33:54.549816 20599 ts_tablet_manager.cc:1428] T 023f28c4dbf541f6a426ecefa09af94a P 9cd11d467628470984d8ebd33bdafd6c: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
W20260926 11:33:54.607093 20318 tablet.cc:2369] T 023f28c4dbf541f6a426ecefa09af94a P 9cd11d467628470984d8ebd33bdafd6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 11:33:54.724583 20580 tablet.cc:2369] T 023f28c4dbf541f6a426ecefa09af94a P fbe5d43e75374517b4462a3bc596977d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:33:54.750160 20603 raft_consensus.cc:482] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 11:33:54.750401 20603 raft_consensus.cc:504] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd11d467628470984d8ebd33bdafd6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46629 } } peers { permanent_uuid: "fbe5d43e75374517b4462a3bc596977d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41137 } } peers { permanent_uuid: "aae445ca50a540609a68b80c428da496" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40463 } }
I20260926 11:33:54.751638 20603 leader_election.cc:288] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cd11d467628470984d8ebd33bdafd6c (127.0.98.1:46629), fbe5d43e75374517b4462a3bc596977d (127.0.98.3:41137)
I20260926 11:33:54.755210 20594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:54.751994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45877 (local address 127.0.98.3:41137)
0926 11:33:54.752124 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:54.752128 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:54.752143 (+ 15us) server_negotiation.cc:407] Connection header received
0926 11:33:54.752268 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:54.752272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:54.752325 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:54.752417 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:54.752682 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.753193 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.754256 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.754666 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.754782 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:54.754843 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:54.755020 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 11:33:54.755087 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 11:33:54.755210 20450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:54.751900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41137 (local address 127.0.98.2:45877)
0926 11:33:54.752038 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:54.752055 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:54.752145 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:54.752435 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:54.752438 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:54.752456 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:54.752576 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:54.752582 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.753306 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.753311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:54.754137 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:54.754146 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.754796 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.754799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:54.754903 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:54.754920 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 11:33:54.755005 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 11:33:54.755075 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":23}
I20260926 11:33:54.755630 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:54.752226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46629 (local address 127.0.98.2:52473)
0926 11:33:54.752545 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:54.752556 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 11:33:54.752674 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:33:54.752942 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:33:54.752945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:33:54.752960 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 11:33:54.753084 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:54.753089 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.753774 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.753779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:54.754559 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:33:54.754567 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.755200 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.755203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:33:54.755301 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:54.755345 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 11:33:54.755425 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 11:33:54.755481 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":1,"thread_start_us":138,"threads_started":1}
I20260926 11:33:54.755664 20592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:54.752033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52473 (local address 127.0.98.1:46629)
0926 11:33:54.752152 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:54.752155 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:54.752640 (+ 485us) server_negotiation.cc:407] Connection header received
0926 11:33:54.752785 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:54.752788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:54.752846 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:54.752932 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 11:33:54.753190 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.753666 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.754677 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.755074 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.755194 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:54.755251 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:54.755451 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 11:33:54.755528 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 11:33:54.755976 20533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "023f28c4dbf541f6a426ecefa09af94a" candidate_uuid: "aae445ca50a540609a68b80c428da496" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe5d43e75374517b4462a3bc596977d" is_pre_election: true
I20260926 11:33:54.756348 20533 raft_consensus.cc:2454] T 023f28c4dbf541f6a426ecefa09af94a P fbe5d43e75374517b4462a3bc596977d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae445ca50a540609a68b80c428da496 in term 0.
I20260926 11:33:54.756356 20271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "023f28c4dbf541f6a426ecefa09af94a" candidate_uuid: "aae445ca50a540609a68b80c428da496" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cd11d467628470984d8ebd33bdafd6c" is_pre_election: true
I20260926 11:33:54.756657 20271 raft_consensus.cc:2454] T 023f28c4dbf541f6a426ecefa09af94a P 9cd11d467628470984d8ebd33bdafd6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae445ca50a540609a68b80c428da496 in term 0.
I20260926 11:33:54.756893 20336 leader_election.cc:302] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aae445ca50a540609a68b80c428da496, fbe5d43e75374517b4462a3bc596977d; no voters:
I20260926 11:33:54.757205 20603 raft_consensus.cc:2781] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 11:33:54.757282 20603 raft_consensus.cc:482] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 11:33:54.757329 20603 raft_consensus.cc:3037] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:33:54.758345 20603 raft_consensus.cc:504] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd11d467628470984d8ebd33bdafd6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46629 } } peers { permanent_uuid: "fbe5d43e75374517b4462a3bc596977d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41137 } } peers { permanent_uuid: "aae445ca50a540609a68b80c428da496" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40463 } }
I20260926 11:33:54.758746 20603 leader_election.cc:288] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496 [CANDIDATE]: Term 1 election: Requested vote from peers 9cd11d467628470984d8ebd33bdafd6c (127.0.98.1:46629), fbe5d43e75374517b4462a3bc596977d (127.0.98.3:41137)
I20260926 11:33:54.759083 20271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "023f28c4dbf541f6a426ecefa09af94a" candidate_uuid: "aae445ca50a540609a68b80c428da496" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cd11d467628470984d8ebd33bdafd6c"
I20260926 11:33:54.759225 20271 raft_consensus.cc:3037] T 023f28c4dbf541f6a426ecefa09af94a P 9cd11d467628470984d8ebd33bdafd6c [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:33:54.759263 20533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "023f28c4dbf541f6a426ecefa09af94a" candidate_uuid: "aae445ca50a540609a68b80c428da496" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbe5d43e75374517b4462a3bc596977d"
I20260926 11:33:54.759368 20533 raft_consensus.cc:3037] T 023f28c4dbf541f6a426ecefa09af94a P fbe5d43e75374517b4462a3bc596977d [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:33:54.760386 20271 raft_consensus.cc:2454] T 023f28c4dbf541f6a426ecefa09af94a P 9cd11d467628470984d8ebd33bdafd6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae445ca50a540609a68b80c428da496 in term 1.
I20260926 11:33:54.760455 20533 raft_consensus.cc:2454] T 023f28c4dbf541f6a426ecefa09af94a P fbe5d43e75374517b4462a3bc596977d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae445ca50a540609a68b80c428da496 in term 1.
I20260926 11:33:54.760756 20335 leader_election.cc:302] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cd11d467628470984d8ebd33bdafd6c, aae445ca50a540609a68b80c428da496; no voters:
I20260926 11:33:54.760974 20603 raft_consensus.cc:2781] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:33:54.761226 20603 raft_consensus.cc:686] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496 [term 1 LEADER]: Becoming Leader. State: Replica: aae445ca50a540609a68b80c428da496, State: Running, Role: LEADER
I20260926 11:33:54.761566 20603 consensus_queue.cc:237] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd11d467628470984d8ebd33bdafd6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46629 } } peers { permanent_uuid: "fbe5d43e75374517b4462a3bc596977d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41137 } } peers { permanent_uuid: "aae445ca50a540609a68b80c428da496" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40463 } }
I20260926 11:33:54.764885 20128 catalog_manager.cc:5495] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496 reported cstate change: term changed from 0 to 1, leader changed from <none> to aae445ca50a540609a68b80c428da496 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "aae445ca50a540609a68b80c428da496" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd11d467628470984d8ebd33bdafd6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbe5d43e75374517b4462a3bc596977d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aae445ca50a540609a68b80c428da496" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40463 } health_report { overall_health: HEALTHY } } }
W20260926 11:33:54.790683 20449 tablet.cc:2369] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:33:54.813813 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:54.810260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52920 (local address 127.0.98.2:40463)
0926 11:33:54.810414 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:54.810419 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:54.810440 (+ 21us) server_negotiation.cc:407] Connection header received
0926 11:33:54.810517 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:54.810519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:54.810573 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:54.810706 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:33:54.810980 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.811481 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.812361 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:54.812580 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:54.812689 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:54.812999 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:33:54.813413 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:33:54.813553 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:54.813594 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 11:33:54.813646 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 11:33:54.819808 20533 raft_consensus.cc:1261] T 023f28c4dbf541f6a426ecefa09af94a P fbe5d43e75374517b4462a3bc596977d [term 1 FOLLOWER]: Refusing update from remote peer aae445ca50a540609a68b80c428da496: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:33:54.819836 20271 raft_consensus.cc:1261] T 023f28c4dbf541f6a426ecefa09af94a P 9cd11d467628470984d8ebd33bdafd6c [term 1 FOLLOWER]: Refusing update from remote peer aae445ca50a540609a68b80c428da496: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:33:54.820529 20606 consensus_queue.cc:1035] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbe5d43e75374517b4462a3bc596977d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:33:54.820750 20603 consensus_queue.cc:1035] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cd11d467628470984d8ebd33bdafd6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:33:54.831476 20610 mvcc.cc:205] Tried to move back new op lower bound from 7333570293014466560 to 7333570292786003968. Current Snapshot: MvccSnapshot[applied={T|T < 7333570293014466560 or (T in {7333570293014466560})}]
I20260926 11:33:54.837071 20129 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:33:54.837226 20129 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 11:33:54.839583 20129 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46e6dfb706834743a3568306f96183a7: Sending DeleteTablet for 3 replicas of tablet 023f28c4dbf541f6a426ecefa09af94a
I20260926 11:33:54.840372 20513 tablet_service.cc:1514] Processing DeleteTablet for tablet 023f28c4dbf541f6a426ecefa09af94a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:33:54 UTC) from {username='slave'} at 127.0.0.1:56372
I20260926 11:33:54.840512 20251 tablet_service.cc:1514] Processing DeleteTablet for tablet 023f28c4dbf541f6a426ecefa09af94a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:33:54 UTC) from {username='slave'} at 127.0.0.1:40956
I20260926 11:33:54.840523 20382 tablet_service.cc:1514] Processing DeleteTablet for tablet 023f28c4dbf541f6a426ecefa09af94a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 11:33:54 UTC) from {username='slave'} at 127.0.0.1:52918
I20260926 11:33:54.841068 20619 tablet_replica.cc:330] T 023f28c4dbf541f6a426ecefa09af94a P fbe5d43e75374517b4462a3bc596977d: stopping tablet replica
I20260926 11:33:54.841307 20620 tablet_replica.cc:330] T 023f28c4dbf541f6a426ecefa09af94a P 9cd11d467628470984d8ebd33bdafd6c: stopping tablet replica
I20260926 11:33:54.841190 20621 tablet_replica.cc:330] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496: stopping tablet replica
I20260926 11:33:54.841681 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 20189
I20260926 11:33:54.842584 20619 raft_consensus.cc:2229] T 023f28c4dbf541f6a426ecefa09af94a P fbe5d43e75374517b4462a3bc596977d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 11:33:54.842861 20621 raft_consensus.cc:2229] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496 [term 1 LEADER]: Raft consensus shutting down.
I20260926 11:33:54.843007 20619 raft_consensus.cc:2258] T 023f28c4dbf541f6a426ecefa09af94a P fbe5d43e75374517b4462a3bc596977d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:33:54.843434 20621 raft_consensus.cc:2258] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 11:33:54.844704 20621 ts_tablet_manager.cc:1905] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 11:33:54.846805 20621 ts_tablet_manager.cc:1918] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:33:54.846913 20621 log.cc:1196] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-1/wal/wals/023f28c4dbf541f6a426ecefa09af94a
I20260926 11:33:54.847208 20621 ts_tablet_manager.cc:1939] T 023f28c4dbf541f6a426ecefa09af94a P aae445ca50a540609a68b80c428da496: Deleting consensus metadata
W20260926 11:33:54.849159 20113 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46629 (error 108)
I20260926 11:33:54.849402 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 20320
I20260926 11:33:54.850795 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:54.850495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46629 (local address 127.0.0.1:40962)
0926 11:33:54.850631 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:54.850705 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 11:33:54.851004 20619 ts_tablet_manager.cc:1905] T 023f28c4dbf541f6a426ecefa09af94a P fbe5d43e75374517b4462a3bc596977d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 11:33:54.851321 20113 catalog_manager.cc:4584] TS 9cd11d467628470984d8ebd33bdafd6c (127.0.98.1:46629): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 023f28c4dbf541f6a426ecefa09af94a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46629: connect: Connection refused (error 111)
I20260926 11:33:54.853197 20619 ts_tablet_manager.cc:1918] T 023f28c4dbf541f6a426ecefa09af94a P fbe5d43e75374517b4462a3bc596977d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 11:33:54.853307 20619 log.cc:1196] T 023f28c4dbf541f6a426ecefa09af94a P fbe5d43e75374517b4462a3bc596977d: Deleting WAL directory at /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790422180210734-392-0/minicluster-data/ts-2/wal/wals/023f28c4dbf541f6a426ecefa09af94a
I20260926 11:33:54.853582 20619 ts_tablet_manager.cc:1939] T 023f28c4dbf541f6a426ecefa09af94a P fbe5d43e75374517b4462a3bc596977d: Deleting consensus metadata
I20260926 11:33:54.854977 20114 catalog_manager.cc:4841] TS fbe5d43e75374517b4462a3bc596977d (127.0.98.3:41137): tablet 023f28c4dbf541f6a426ecefa09af94a (table test-table [id=dea76d748e554eccabcdcdeddef6db46]) successfully deleted
I20260926 11:33:54.856494 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 20451
I20260926 11:33:54.861011 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:54.860704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40463 (local address 127.0.0.1:52928)
0926 11:33:54.860879 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 11:33:54.860936 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260926 11:33:54.862196 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 20098
2026-09-26T11:33:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1208 ms)
[----------] 6 tests from EncryptionPolicyTest (10317 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:33:54 dist-test-slave-mrck krb5kdc[20625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:33:54 dist-test-slave-mrck krb5kdc[20625](info): set up 2 sockets
Sep 26 11:33:54 dist-test-slave-mrck krb5kdc[20625](info): commencing operation
krb5kdc: starting...
W20260926 11:33:55.903949 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:33:55 dist-test-slave-mrck krb5kdc[20625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:33:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:33:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 11:33:56.027088 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35203
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:33:56.138319 20638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:56.138649 20638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:56.138697 20638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:56.142920 20638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:33:56.143006 20638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:56.143028 20638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:56.143045 20638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:33:56.143061 20638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:33:56.148643 20638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35203
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:56.149798 20638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:56.150876 20638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:33:56.159471 20644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:56.159492 20646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:56.159791 20643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:56.160043 20638 server_base.cc:1034] running on GCE node
I20260926 11:33:56.160634 20638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:56.161738 20638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:56.162943 20638 hybrid_clock.cc:648] HybridClock initialized: now 1790422436162929 us; error 26 us; skew 500 ppm
Sep 26 11:33:56 dist-test-slave-mrck krb5kdc[20625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:33:56.169246 20638 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 11:33:56.170257 20638 webserver.cc:460] Webserver started at http://127.0.0.1:38629/ using document root <none> and password file <none>
I20260926 11:33:56.170795 20638 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:56.170854 20638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:56.171041 20638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:33:56.173121 20638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "c0406af3ff9a42c387af0c8c42b03355"
format_stamp: "Formatted at 2026-09-26 11:33:56 on dist-test-slave-mrck"
server_key: "3ed935578f224f64ca84169e30def32b"
server_key_iv: "a5b5d76b644495f22346a754eced4a50"
server_key_version: "encryptionkey@0"
I20260926 11:33:56.173629 20638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "c0406af3ff9a42c387af0c8c42b03355"
format_stamp: "Formatted at 2026-09-26 11:33:56 on dist-test-slave-mrck"
server_key: "3ed935578f224f64ca84169e30def32b"
server_key_iv: "a5b5d76b644495f22346a754eced4a50"
server_key_version: "encryptionkey@0"
I20260926 11:33:56.177240 20638 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:33:56.179587 20653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:56.180876 20638 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:33:56.180982 20638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "c0406af3ff9a42c387af0c8c42b03355"
format_stamp: "Formatted at 2026-09-26 11:33:56 on dist-test-slave-mrck"
server_key: "3ed935578f224f64ca84169e30def32b"
server_key_iv: "a5b5d76b644495f22346a754eced4a50"
server_key_version: "encryptionkey@0"
I20260926 11:33:56.181072 20638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:56.202719 20638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:56.205801 20638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:56.205971 20638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:56.212961 20638 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35203
I20260926 11:33:56.212976 20705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35203 every 8 connection(s)
I20260926 11:33:56.213960 20638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:33:56.216718 20706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:33:56.222386 20706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0406af3ff9a42c387af0c8c42b03355: Bootstrap starting.
I20260926 11:33:56.223762 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 20638
I20260926 11:33:56.223904 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:33:56.224212 392 external_mini_cluster.cc:1428] Setting key 14f31f7da508654ee0ae3cb41af4d901
I20260926 11:33:56.225472 20706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0406af3ff9a42c387af0c8c42b03355: Neither blocks nor log segments found. Creating new log.
I20260926 11:33:56.226270 20706 log.cc:824] T 00000000000000000000000000000000 P c0406af3ff9a42c387af0c8c42b03355: Log is configured to *not* fsync() on all Append() calls
I20260926 11:33:56.228674 20706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0406af3ff9a42c387af0c8c42b03355: No bootstrap required, opened a new log
I20260926 11:33:56.233323 20706 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0406af3ff9a42c387af0c8c42b03355 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0406af3ff9a42c387af0c8c42b03355" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35203 } }
I20260926 11:33:56.233603 20706 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0406af3ff9a42c387af0c8c42b03355 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:33:56.233695 20706 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0406af3ff9a42c387af0c8c42b03355 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0406af3ff9a42c387af0c8c42b03355, State: Initialized, Role: FOLLOWER
Sep 26 11:33:56 dist-test-slave-mrck krb5kdc[20625](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 11:33:56.234280 20706 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0406af3ff9a42c387af0c8c42b03355 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0406af3ff9a42c387af0c8c42b03355" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35203 } }
I20260926 11:33:56.234465 20706 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0406af3ff9a42c387af0c8c42b03355 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:33:56.234541 20706 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0406af3ff9a42c387af0c8c42b03355 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:33:56.234651 20706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0406af3ff9a42c387af0c8c42b03355 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:33:56.235925 20706 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0406af3ff9a42c387af0c8c42b03355 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0406af3ff9a42c387af0c8c42b03355" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35203 } }
I20260926 11:33:56.236325 20706 leader_election.cc:302] T 00000000000000000000000000000000 P c0406af3ff9a42c387af0c8c42b03355 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0406af3ff9a42c387af0c8c42b03355; no voters:
I20260926 11:33:56.236681 20706 leader_election.cc:288] T 00000000000000000000000000000000 P c0406af3ff9a42c387af0c8c42b03355 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:33:56.236816 20711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0406af3ff9a42c387af0c8c42b03355 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:33:56.237044 20711 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0406af3ff9a42c387af0c8c42b03355 [term 1 LEADER]: Becoming Leader. State: Replica: c0406af3ff9a42c387af0c8c42b03355, State: Running, Role: LEADER
I20260926 11:33:56.237499 20711 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0406af3ff9a42c387af0c8c42b03355 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0406af3ff9a42c387af0c8c42b03355" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35203 } }
I20260926 11:33:56.238128 20706 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0406af3ff9a42c387af0c8c42b03355 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:33:56.239373 20712 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0406af3ff9a42c387af0c8c42b03355 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0406af3ff9a42c387af0c8c42b03355" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0406af3ff9a42c387af0c8c42b03355" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35203 } } }
I20260926 11:33:56.239655 20712 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0406af3ff9a42c387af0c8c42b03355 [sys.catalog]: This master's current role is: LEADER
I20260926 11:33:56.239655 20713 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0406af3ff9a42c387af0c8c42b03355 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0406af3ff9a42c387af0c8c42b03355. Latest consensus state: current_term: 1 leader_uuid: "c0406af3ff9a42c387af0c8c42b03355" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0406af3ff9a42c387af0c8c42b03355" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35203 } } }
I20260926 11:33:56.239859 20713 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0406af3ff9a42c387af0c8c42b03355 [sys.catalog]: This master's current role is: LEADER
I20260926 11:33:56.241708 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:56.225446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47287 (local address 127.0.0.1:35203)
0926 11:33:56.225905 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:56.225923 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:56.225959 (+ 36us) server_negotiation.cc:407] Connection header received
0926 11:33:56.226796 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:56.226823 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:56.227263 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:56.227569 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:56.228515 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:56.229549 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:56.230739 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:56.231253 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:56.231391 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:56.234631 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:56.234638 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:56.234661 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:33:56.234694 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:56.236919 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:56.237614 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:56.237621 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:56.237626 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:56.237776 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:56.238171 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:56.238175 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:56.238176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:56.238757 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:56.239004 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:56.240378 (+ 1374us) server_negotiation.cc:299] Negotiation successful
0926 11:33:56.240685 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":148,"threads_started":1}
W20260926 11:33:56.244840 20726 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0406af3ff9a42c387af0c8c42b03355: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:33:56.244956 20726 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:33:56.245164 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:33:56.245187 20723 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:33:56.245903 20723 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:33:56.252015 20723 catalog_manager.cc:1347] Generated new cluster ID: e9293e047d844790a8c3dbb749dd4856
I20260926 11:33:56.252174 20723 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:33:56.256664 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:56.247620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60411 (local address 127.0.0.1:35203)
0926 11:33:56.247757 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:56.247761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:56.247776 (+ 15us) server_negotiation.cc:407] Connection header received
0926 11:33:56.247883 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:56.247886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:56.247930 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:56.248011 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:56.248851 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:56.249979 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:56.250958 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:56.251211 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:56.251327 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:56.253027 (+ 1700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:56.253031 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:56.253034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:33:56.253060 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:56.254760 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:56.255308 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:56.255312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:56.255315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:56.255404 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:33:56.255758 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:33:56.255762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:33:56.255764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:33:56.255992 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:56.256155 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:56.256405 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 11:33:56.256505 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 11:33:56.266047 20723 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:33:56.266995 20723 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:33:56.282480 20723 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0406af3ff9a42c387af0c8c42b03355: Generated new TSK 0
I20260926 11:33:56.283401 20723 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:33:56.291160 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 20638
W20260926 11:33:56.296240 20623 connection.cc:535] client connection to 127.0.0.1:35203 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35203): Connection reset by peer (error 104)
2026-09-26T11:33:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1427 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:33:56 dist-test-slave-mrck krb5kdc[20737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:33:56 dist-test-slave-mrck krb5kdc[20737](info): set up 2 sockets
Sep 26 11:33:56 dist-test-slave-mrck krb5kdc[20737](info): commencing operation
krb5kdc: starting...
W20260926 11:33:57.331939 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:33:57 dist-test-slave-mrck krb5kdc[20737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:33:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:33:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 11:33:57.452086 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42147
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:33:57.561097 20750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:57.561447 20750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:57.561532 20750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:57.565688 20750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:33:57.565785 20750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:57.565821 20750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:57.565862 20750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:33:57.565896 20750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:33:57.571266 20750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42147
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:57.572458 20750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:57.573483 20750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 11:33:57.581914 20750 server_base.cc:1034] running on GCE node
W20260926 11:33:57.581867 20756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:57.581868 20755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:57.581873 20758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:57.582836 20750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:57.584051 20750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:57.585249 20750 hybrid_clock.cc:648] HybridClock initialized: now 1790422437585235 us; error 30 us; skew 500 ppm
Sep 26 11:33:57 dist-test-slave-mrck krb5kdc[20737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:33:57.588055 20750 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 11:33:57.588954 20750 webserver.cc:460] Webserver started at http://127.0.0.1:33537/ using document root <none> and password file <none>
I20260926 11:33:57.589526 20750 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:57.589617 20750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:57.589861 20750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:33:57.591969 20750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "17559f5a35184027af3d3bab789c47e6"
format_stamp: "Formatted at 2026-09-26 11:33:57 on dist-test-slave-mrck"
server_key: "5942c47f2cafde8bbb9062e8494332be"
server_key_iv: "cb69381645d3d889ba130f50f3fb4b0c"
server_key_version: "encryptionkey@0"
I20260926 11:33:57.592531 20750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "17559f5a35184027af3d3bab789c47e6"
format_stamp: "Formatted at 2026-09-26 11:33:57 on dist-test-slave-mrck"
server_key: "5942c47f2cafde8bbb9062e8494332be"
server_key_iv: "cb69381645d3d889ba130f50f3fb4b0c"
server_key_version: "encryptionkey@0"
I20260926 11:33:57.596331 20750 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:33:57.598801 20765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:57.600227 20750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:33:57.600385 20750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "17559f5a35184027af3d3bab789c47e6"
format_stamp: "Formatted at 2026-09-26 11:33:57 on dist-test-slave-mrck"
server_key: "5942c47f2cafde8bbb9062e8494332be"
server_key_iv: "cb69381645d3d889ba130f50f3fb4b0c"
server_key_version: "encryptionkey@0"
I20260926 11:33:57.600512 20750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:57.637707 20750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:57.640941 20750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:57.641176 20750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:57.648196 20750 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42147
I20260926 11:33:57.648219 20817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42147 every 8 connection(s)
I20260926 11:33:57.649205 20750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:33:57.652124 20818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:33:57.657598 20818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17559f5a35184027af3d3bab789c47e6: Bootstrap starting.
I20260926 11:33:57.658540 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 20750
I20260926 11:33:57.658676 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:33:57.659032 392 external_mini_cluster.cc:1428] Setting key 7368ee550685f4a191ba48c263691894
I20260926 11:33:57.660619 20818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17559f5a35184027af3d3bab789c47e6: Neither blocks nor log segments found. Creating new log.
I20260926 11:33:57.661397 20818 log.cc:824] T 00000000000000000000000000000000 P 17559f5a35184027af3d3bab789c47e6: Log is configured to *not* fsync() on all Append() calls
I20260926 11:33:57.663412 20818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17559f5a35184027af3d3bab789c47e6: No bootstrap required, opened a new log
I20260926 11:33:57.667212 20818 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17559f5a35184027af3d3bab789c47e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17559f5a35184027af3d3bab789c47e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42147 } }
I20260926 11:33:57.667534 20818 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17559f5a35184027af3d3bab789c47e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:33:57.667614 20818 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17559f5a35184027af3d3bab789c47e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17559f5a35184027af3d3bab789c47e6, State: Initialized, Role: FOLLOWER
I20260926 11:33:57.668113 20818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17559f5a35184027af3d3bab789c47e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17559f5a35184027af3d3bab789c47e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42147 } }
I20260926 11:33:57.668241 20818 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17559f5a35184027af3d3bab789c47e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:33:57.668330 20818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17559f5a35184027af3d3bab789c47e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:33:57.668454 20818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17559f5a35184027af3d3bab789c47e6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:33:57.668599 20821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:57.660218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49707 (local address 127.0.0.1:42147)
0926 11:33:57.660719 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:57.660739 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:57.660798 (+ 59us) server_negotiation.cc:407] Connection header received
0926 11:33:57.661518 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:57.661549 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:57.662002 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:57.662399 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:57.662819 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:57.663916 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:57.664820 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:57.665305 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:57.665497 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:57.666516 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:57.666521 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:57.666544 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:33:57.666966 (+ 422us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:57.667022 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:33:57.667050 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:57.667195 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:57.667375 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 11:33:57.667519 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":145,"threads_started":1}
I20260926 11:33:57.669454 20818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17559f5a35184027af3d3bab789c47e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17559f5a35184027af3d3bab789c47e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42147 } }
I20260926 11:33:57.669849 20818 leader_election.cc:302] T 00000000000000000000000000000000 P 17559f5a35184027af3d3bab789c47e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17559f5a35184027af3d3bab789c47e6; no voters:
I20260926 11:33:57.670183 20818 leader_election.cc:288] T 00000000000000000000000000000000 P 17559f5a35184027af3d3bab789c47e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:33:57.670297 20823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17559f5a35184027af3d3bab789c47e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:33:57.670563 20823 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17559f5a35184027af3d3bab789c47e6 [term 1 LEADER]: Becoming Leader. State: Replica: 17559f5a35184027af3d3bab789c47e6, State: Running, Role: LEADER
I20260926 11:33:57.670926 20823 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17559f5a35184027af3d3bab789c47e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17559f5a35184027af3d3bab789c47e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42147 } }
I20260926 11:33:57.671504 20818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17559f5a35184027af3d3bab789c47e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:33:57.672019 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:33:57.672713 20824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17559f5a35184027af3d3bab789c47e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17559f5a35184027af3d3bab789c47e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17559f5a35184027af3d3bab789c47e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42147 } } }
I20260926 11:33:57.672857 20824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17559f5a35184027af3d3bab789c47e6 [sys.catalog]: This master's current role is: LEADER
I20260926 11:33:57.673208 20825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17559f5a35184027af3d3bab789c47e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17559f5a35184027af3d3bab789c47e6. Latest consensus state: current_term: 1 leader_uuid: "17559f5a35184027af3d3bab789c47e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17559f5a35184027af3d3bab789c47e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42147 } } }
I20260926 11:33:57.673343 20825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17559f5a35184027af3d3bab789c47e6 [sys.catalog]: This master's current role is: LEADER
W20260926 11:33:57.676769 20842 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 17559f5a35184027af3d3bab789c47e6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:33:57.676869 20842 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:33:57.676929 20832 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:33:57.677748 20832 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:33:57.679605 20821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:57.674802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59403 (local address 127.0.0.1:42147)
0926 11:33:57.674936 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:57.674941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:57.675121 (+ 180us) server_negotiation.cc:407] Connection header received
0926 11:33:57.675277 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:57.675281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:57.675333 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:57.675429 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:57.675719 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:57.676539 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:57.677661 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:33:57.678003 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:33:57.678118 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:33:57.679086 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:57.679090 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:57.679093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:33:57.679108 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:57.679136 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:33:57.679149 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:57.679245 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:57.679425 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 11:33:57.679462 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 11:33:57.683449 20832 catalog_manager.cc:1347] Generated new cluster ID: cc88b364772346248441e1b4b06edeae
I20260926 11:33:57.683511 20832 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:33:57.691512 20832 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:33:57.692780 20832 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:33:57.700798 20832 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17559f5a35184027af3d3bab789c47e6: Generated new TSK 0
I20260926 11:33:57.701644 20832 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:33:57.707741 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 20750
W20260926 11:33:57.712584 20735 connection.cc:535] client connection to 127.0.0.1:42147 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42147): Connection reset by peer (error 104)
2026-09-26T11:33:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:33:57 dist-test-slave-mrck krb5kdc[20849](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:33:57 dist-test-slave-mrck krb5kdc[20849](info): set up 2 sockets
Sep 26 11:33:57 dist-test-slave-mrck krb5kdc[20849](info): commencing operation
krb5kdc: starting...
W20260926 11:33:58.740401 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:33:58 dist-test-slave-mrck krb5kdc[20849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:33:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:33:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 11:33:58.859110 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38551
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:33:58.969851 20862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:33:58.970127 20862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:33:58.970170 20862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:33:58.974200 20862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:33:58.974262 20862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:33:58.974282 20862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:33:58.974298 20862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:33:58.974313 20862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:33:58.979605 20862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38551
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:33:58.980954 20862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:33:58.982039 20862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:33:58.990692 20867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:58.990710 20868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:33:58.990761 20870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:33:58.990857 20862 server_base.cc:1034] running on GCE node
I20260926 11:33:58.991432 20862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:33:58.992620 20862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:33:58.993892 20862 hybrid_clock.cc:648] HybridClock initialized: now 1790422438993862 us; error 43 us; skew 500 ppm
Sep 26 11:33:58 dist-test-slave-mrck krb5kdc[20849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:33:58.996963 20862 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 11:33:58.997838 20862 webserver.cc:460] Webserver started at http://127.0.0.1:40591/ using document root <none> and password file <none>
I20260926 11:33:58.998404 20862 fs_manager.cc:359] Metadata directory not provided
I20260926 11:33:58.998494 20862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:33:58.998725 20862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:33:59.000833 20862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "ad908b8e80d842d089d49430eed75e42"
format_stamp: "Formatted at 2026-09-26 11:33:58 on dist-test-slave-mrck"
server_key: "d2381f844746f0458842b917298f00ca"
server_key_iv: "0574e329c36987de0de0d3908f6cd609"
server_key_version: "encryptionkey@0"
I20260926 11:33:59.001406 20862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "ad908b8e80d842d089d49430eed75e42"
format_stamp: "Formatted at 2026-09-26 11:33:58 on dist-test-slave-mrck"
server_key: "d2381f844746f0458842b917298f00ca"
server_key_iv: "0574e329c36987de0de0d3908f6cd609"
server_key_version: "encryptionkey@0"
I20260926 11:33:59.005168 20862 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:33:59.007640 20877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:33:59.008921 20862 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:33:59.009073 20862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "ad908b8e80d842d089d49430eed75e42"
format_stamp: "Formatted at 2026-09-26 11:33:58 on dist-test-slave-mrck"
server_key: "d2381f844746f0458842b917298f00ca"
server_key_iv: "0574e329c36987de0de0d3908f6cd609"
server_key_version: "encryptionkey@0"
I20260926 11:33:59.009189 20862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:33:59.013726 20862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:33:59.017059 20862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:33:59.017248 20862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:33:59.024325 20862 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38551
I20260926 11:33:59.024385 20929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38551 every 8 connection(s)
I20260926 11:33:59.025319 20862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:33:59.028259 20930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:33:59.033699 20930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad908b8e80d842d089d49430eed75e42: Bootstrap starting.
I20260926 11:33:59.035178 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 20862
I20260926 11:33:59.035331 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:33:59.035626 392 external_mini_cluster.cc:1428] Setting key f81235ae6d6cda6fa268933d03a52ae0
I20260926 11:33:59.036341 20930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad908b8e80d842d089d49430eed75e42: Neither blocks nor log segments found. Creating new log.
I20260926 11:33:59.037144 20930 log.cc:824] T 00000000000000000000000000000000 P ad908b8e80d842d089d49430eed75e42: Log is configured to *not* fsync() on all Append() calls
I20260926 11:33:59.039269 20930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad908b8e80d842d089d49430eed75e42: No bootstrap required, opened a new log
I20260926 11:33:59.040609 20933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:59.036749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34741 (local address 127.0.0.1:38551)
0926 11:33:59.037231 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:59.037251 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:59.037298 (+ 47us) server_negotiation.cc:407] Connection header received
0926 11:33:59.037918 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:59.037934 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:59.038231 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:59.038506 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:59.039347 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:59.039355 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:59.039375 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:33:59.039394 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:59.039447 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:33:59.039467 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:59.039556 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:59.039699 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 11:33:59.039893 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":155,"threads_started":1}
I20260926 11:33:59.043051 20930 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad908b8e80d842d089d49430eed75e42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad908b8e80d842d089d49430eed75e42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38551 } }
I20260926 11:33:59.043289 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:33:59.043315 20930 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad908b8e80d842d089d49430eed75e42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:33:59.043401 20930 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad908b8e80d842d089d49430eed75e42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad908b8e80d842d089d49430eed75e42, State: Initialized, Role: FOLLOWER
I20260926 11:33:59.043900 20930 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad908b8e80d842d089d49430eed75e42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad908b8e80d842d089d49430eed75e42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38551 } }
I20260926 11:33:59.044034 20930 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad908b8e80d842d089d49430eed75e42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:33:59.044113 20930 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad908b8e80d842d089d49430eed75e42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:33:59.044212 20930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad908b8e80d842d089d49430eed75e42 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:33:59.045162 20930 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad908b8e80d842d089d49430eed75e42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad908b8e80d842d089d49430eed75e42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38551 } }
I20260926 11:33:59.045557 20930 leader_election.cc:302] T 00000000000000000000000000000000 P ad908b8e80d842d089d49430eed75e42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad908b8e80d842d089d49430eed75e42; no voters:
I20260926 11:33:59.045920 20930 leader_election.cc:288] T 00000000000000000000000000000000 P ad908b8e80d842d089d49430eed75e42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:33:59.046011 20940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad908b8e80d842d089d49430eed75e42 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:33:59.046247 20940 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad908b8e80d842d089d49430eed75e42 [term 1 LEADER]: Becoming Leader. State: Replica: ad908b8e80d842d089d49430eed75e42, State: Running, Role: LEADER
I20260926 11:33:59.046561 20940 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad908b8e80d842d089d49430eed75e42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad908b8e80d842d089d49430eed75e42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38551 } }
I20260926 11:33:59.047312 20930 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad908b8e80d842d089d49430eed75e42 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:33:59.047331 20933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:33:59.045440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39487 (local address 127.0.0.1:38551)
0926 11:33:59.045553 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 11:33:59.045557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:33:59.045740 (+ 183us) server_negotiation.cc:407] Connection header received
0926 11:33:59.045854 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:33:59.045857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:33:59.045888 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:33:59.045963 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:33:59.046958 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:33:59.046962 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:33:59.046965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:33:59.046976 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:33:59.047004 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:33:59.047017 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:33:59.047101 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 11:33:59.047195 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 11:33:59.047229 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 11:33:59.048362 20942 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad908b8e80d842d089d49430eed75e42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad908b8e80d842d089d49430eed75e42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad908b8e80d842d089d49430eed75e42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38551 } } }
I20260926 11:33:59.048485 20942 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad908b8e80d842d089d49430eed75e42 [sys.catalog]: This master's current role is: LEADER
I20260926 11:33:59.048804 20941 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad908b8e80d842d089d49430eed75e42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad908b8e80d842d089d49430eed75e42. Latest consensus state: current_term: 1 leader_uuid: "ad908b8e80d842d089d49430eed75e42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad908b8e80d842d089d49430eed75e42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38551 } } }
I20260926 11:33:59.048904 20941 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad908b8e80d842d089d49430eed75e42 [sys.catalog]: This master's current role is: LEADER
I20260926 11:33:59.048875 20949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:33:59.051699 20949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:33:59.057160 20949 catalog_manager.cc:1347] Generated new cluster ID: e3c9507c76364ecb976fff89ea551490
I20260926 11:33:59.057268 20949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:33:59.072492 20949 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:33:59.072834 20949 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:33:59.080374 20949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad908b8e80d842d089d49430eed75e42: Generated new TSK 0
I20260926 11:33:59.081118 20949 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:33:59.092573 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 20862
2026-09-26T11:33:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:33:59 dist-test-slave-mrck krb5kdc[20961](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:33:59 dist-test-slave-mrck krb5kdc[20961](info): set up 2 sockets
Sep 26 11:33:59 dist-test-slave-mrck krb5kdc[20961](info): commencing operation
krb5kdc: starting...
W20260926 11:34:00.123993 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:34:00 dist-test-slave-mrck krb5kdc[20961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:34:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:34:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 11:34:00.241007 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45891
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:34:00.350375 20974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:34:00.350656 20974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:34:00.350746 20974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:34:00.354893 20974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:34:00.354996 20974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:34:00.355036 20974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:34:00.355095 20974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:34:00.355135 20974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:34:00.360517 20974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:34:00.361580 20974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:34:00.362610 20974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 11:34:00.371284 20974 server_base.cc:1034] running on GCE node
W20260926 11:34:00.371235 20979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:34:00.371235 20980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:34:00.371261 20982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:34:00.372100 20974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:34:00.373198 20974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:34:00.374399 20974 hybrid_clock.cc:648] HybridClock initialized: now 1790422440374377 us; error 32 us; skew 500 ppm
Sep 26 11:34:00 dist-test-slave-mrck krb5kdc[20961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:34:00.377269 20974 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 11:34:00.378125 20974 webserver.cc:460] Webserver started at http://127.0.0.1:40293/ using document root <none> and password file <none>
I20260926 11:34:00.378676 20974 fs_manager.cc:359] Metadata directory not provided
I20260926 11:34:00.378734 20974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:34:00.378902 20974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:34:00.380916 20974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "17be03712d2d4d7fb4f8fa897a72212e"
format_stamp: "Formatted at 2026-09-26 11:34:00 on dist-test-slave-mrck"
server_key: "dd054136bd384b99c576d2c390db4744"
server_key_iv: "1d4f6a16a825bd50b92f5889cb187eca"
server_key_version: "encryptionkey@0"
I20260926 11:34:00.381413 20974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "17be03712d2d4d7fb4f8fa897a72212e"
format_stamp: "Formatted at 2026-09-26 11:34:00 on dist-test-slave-mrck"
server_key: "dd054136bd384b99c576d2c390db4744"
server_key_iv: "1d4f6a16a825bd50b92f5889cb187eca"
server_key_version: "encryptionkey@0"
I20260926 11:34:00.385187 20974 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 11:34:00.387554 20989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:34:00.388927 20974 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:34:00.389117 20974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "17be03712d2d4d7fb4f8fa897a72212e"
format_stamp: "Formatted at 2026-09-26 11:34:00 on dist-test-slave-mrck"
server_key: "dd054136bd384b99c576d2c390db4744"
server_key_iv: "1d4f6a16a825bd50b92f5889cb187eca"
server_key_version: "encryptionkey@0"
I20260926 11:34:00.389241 20974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:34:00.408703 20974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:34:00.412200 20974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:34:00.412421 20974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:34:00.419906 20974 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45891
I20260926 11:34:00.419955 21041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45891 every 8 connection(s)
I20260926 11:34:00.420917 20974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:34:00.423589 21042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:34:00.426815 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 20974
I20260926 11:34:00.426954 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:34:00.427268 392 external_mini_cluster.cc:1428] Setting key f72f6b1c971261b3ef5cf8e9baf16d6e
I20260926 11:34:00.429507 21042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17be03712d2d4d7fb4f8fa897a72212e: Bootstrap starting.
I20260926 11:34:00.431849 21042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17be03712d2d4d7fb4f8fa897a72212e: Neither blocks nor log segments found. Creating new log.
I20260926 11:34:00.432544 21042 log.cc:824] T 00000000000000000000000000000000 P 17be03712d2d4d7fb4f8fa897a72212e: Log is configured to *not* fsync() on all Append() calls
I20260926 11:34:00.434240 21042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17be03712d2d4d7fb4f8fa897a72212e: No bootstrap required, opened a new log
Sep 26 11:34:00 dist-test-slave-mrck krb5kdc[20961](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 11:34:00.438320 21042 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17be03712d2d4d7fb4f8fa897a72212e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17be03712d2d4d7fb4f8fa897a72212e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45891 } }
I20260926 11:34:00.438525 21042 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17be03712d2d4d7fb4f8fa897a72212e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:34:00.438557 21042 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17be03712d2d4d7fb4f8fa897a72212e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17be03712d2d4d7fb4f8fa897a72212e, State: Initialized, Role: FOLLOWER
I20260926 11:34:00.439052 21042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17be03712d2d4d7fb4f8fa897a72212e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17be03712d2d4d7fb4f8fa897a72212e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45891 } }
I20260926 11:34:00.439198 21042 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17be03712d2d4d7fb4f8fa897a72212e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:34:00.439286 21042 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17be03712d2d4d7fb4f8fa897a72212e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:34:00.439409 21042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17be03712d2d4d7fb4f8fa897a72212e [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:34:00.440347 21042 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17be03712d2d4d7fb4f8fa897a72212e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17be03712d2d4d7fb4f8fa897a72212e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45891 } }
I20260926 11:34:00.440711 21042 leader_election.cc:302] T 00000000000000000000000000000000 P 17be03712d2d4d7fb4f8fa897a72212e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17be03712d2d4d7fb4f8fa897a72212e; no voters:
I20260926 11:34:00.441026 21042 leader_election.cc:288] T 00000000000000000000000000000000 P 17be03712d2d4d7fb4f8fa897a72212e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:34:00.441141 21047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17be03712d2d4d7fb4f8fa897a72212e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:34:00.441341 21047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17be03712d2d4d7fb4f8fa897a72212e [term 1 LEADER]: Becoming Leader. State: Replica: 17be03712d2d4d7fb4f8fa897a72212e, State: Running, Role: LEADER
I20260926 11:34:00.441704 21047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17be03712d2d4d7fb4f8fa897a72212e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17be03712d2d4d7fb4f8fa897a72212e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45891 } }
I20260926 11:34:00.442139 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:00.428676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36987 (local address 127.0.0.1:45891)
0926 11:34:00.429087 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:00.429102 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:00.429137 (+ 35us) server_negotiation.cc:407] Connection header received
0926 11:34:00.429829 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:00.429847 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:00.430189 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:00.430609 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:00.431509 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:00.432677 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:00.433504 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:00.434000 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:00.434171 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:00.437016 (+ 2845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:00.437021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:00.437034 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:00.437059 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:00.438782 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:00.439301 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:00.439308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:00.439316 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:00.439453 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:00.439859 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:00.439862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:00.439864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:00.440257 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:00.440416 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:00.440706 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 11:34:00.441189 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":123,"threads_started":1}
I20260926 11:34:00.442304 21042 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17be03712d2d4d7fb4f8fa897a72212e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:34:00.442878 21047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17be03712d2d4d7fb4f8fa897a72212e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17be03712d2d4d7fb4f8fa897a72212e. Latest consensus state: current_term: 1 leader_uuid: "17be03712d2d4d7fb4f8fa897a72212e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17be03712d2d4d7fb4f8fa897a72212e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45891 } } }
I20260926 11:34:00.443040 21047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17be03712d2d4d7fb4f8fa897a72212e [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:00.443284 21048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17be03712d2d4d7fb4f8fa897a72212e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17be03712d2d4d7fb4f8fa897a72212e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17be03712d2d4d7fb4f8fa897a72212e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45891 } } }
I20260926 11:34:00.443382 21048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17be03712d2d4d7fb4f8fa897a72212e [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:00.443452 21052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:34:00.445073 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:34:00.447008 21052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:34:00.453647 21052 catalog_manager.cc:1347] Generated new cluster ID: aa40a751f53d4c7aa7b12b5a53944cee
I20260926 11:34:00.453723 21052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:34:00.455199 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:00.447022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56595 (local address 127.0.0.1:45891)
0926 11:34:00.447134 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:00.447139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:00.447336 (+ 197us) server_negotiation.cc:407] Connection header received
0926 11:34:00.447473 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:00.447477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:00.447522 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:00.447603 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:00.448514 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:00.449220 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:00.450001 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:00.450268 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:00.450379 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:00.451687 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:00.451691 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:00.451694 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:00.451717 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:00.453424 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:00.453973 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:00.453977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:00.453979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:00.454066 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:00.454365 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:00.454368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:00.454370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:00.454590 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:00.454705 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:00.454946 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 11:34:00.455056 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 11:34:00.465967 21052 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:34:00.466940 21052 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:34:00.474987 21052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17be03712d2d4d7fb4f8fa897a72212e: Generated new TSK 0
I20260926 11:34:00.475853 21052 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:34:00.480727 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 20974
2026-09-26T11:34:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1387 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:34:00 dist-test-slave-mrck krb5kdc[21073](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:34:00 dist-test-slave-mrck krb5kdc[21073](info): set up 2 sockets
Sep 26 11:34:00 dist-test-slave-mrck krb5kdc[21073](info): commencing operation
krb5kdc: starting...
W20260926 11:34:01.531927 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:34:01 dist-test-slave-mrck krb5kdc[21073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:34:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:34:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 11:34:01.654378 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44399
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:34:01.768285 21086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:34:01.768599 21086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:34:01.768646 21086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:34:01.772974 21086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:34:01.773046 21086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:34:01.773067 21086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:34:01.773084 21086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:34:01.773100 21086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:34:01.778592 21086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:34:01.779819 21086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:34:01.780895 21086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:34:01.790066 21091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:34:01.790030 21086 server_base.cc:1034] running on GCE node
W20260926 11:34:01.789978 21092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:34:01.789997 21094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:34:01.790915 21086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:34:01.792222 21086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:34:01.793447 21086 hybrid_clock.cc:648] HybridClock initialized: now 1790422441793413 us; error 47 us; skew 500 ppm
Sep 26 11:34:01 dist-test-slave-mrck krb5kdc[21073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:34:01.802810 21086 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 11:34:01.803979 21086 webserver.cc:460] Webserver started at http://127.0.0.1:36843/ using document root <none> and password file <none>
I20260926 11:34:01.804584 21086 fs_manager.cc:359] Metadata directory not provided
I20260926 11:34:01.804690 21086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:34:01.804934 21086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:34:01.807101 21086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "748d80d4fd2743d9bc55cf385cab3621"
format_stamp: "Formatted at 2026-09-26 11:34:01 on dist-test-slave-mrck"
server_key: "cdcbf1443cd47bbaef16340c2de65bea"
server_key_iv: "637a19c76e66201bb8174cb881738ec7"
server_key_version: "encryptionkey@0"
I20260926 11:34:01.807706 21086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "748d80d4fd2743d9bc55cf385cab3621"
format_stamp: "Formatted at 2026-09-26 11:34:01 on dist-test-slave-mrck"
server_key: "cdcbf1443cd47bbaef16340c2de65bea"
server_key_iv: "637a19c76e66201bb8174cb881738ec7"
server_key_version: "encryptionkey@0"
I20260926 11:34:01.811650 21086 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 11:34:01.814199 21101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:34:01.815447 21086 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:34:01.815603 21086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "748d80d4fd2743d9bc55cf385cab3621"
format_stamp: "Formatted at 2026-09-26 11:34:01 on dist-test-slave-mrck"
server_key: "cdcbf1443cd47bbaef16340c2de65bea"
server_key_iv: "637a19c76e66201bb8174cb881738ec7"
server_key_version: "encryptionkey@0"
I20260926 11:34:01.815738 21086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:34:01.838058 21086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:34:01.841138 21086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:34:01.841369 21086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:34:01.848577 21086 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44399
I20260926 11:34:01.848598 21153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44399 every 8 connection(s)
I20260926 11:34:01.849555 21086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:34:01.851274 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 21086
I20260926 11:34:01.851433 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:34:01.851775 392 external_mini_cluster.cc:1428] Setting key e7e1db6e16fe5190c53c1e2607cc71c0
I20260926 11:34:01.852676 21154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:34:01.858314 21154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 748d80d4fd2743d9bc55cf385cab3621: Bootstrap starting.
I20260926 11:34:01.860603 21157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:01.853206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54669 (local address 127.0.0.1:44399)
0926 11:34:01.853652 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:01.853666 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:01.853700 (+ 34us) server_negotiation.cc:407] Connection header received
0926 11:34:01.854333 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:01.854349 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:01.854692 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:01.855029 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:01.855432 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:01.856466 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:01.857375 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:01.857856 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:01.858037 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:01.858959 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:01.858964 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:01.858989 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:34:01.859364 (+ 375us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:01.859434 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:34:01.859456 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:01.859565 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:01.859679 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 11:34:01.859910 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":152,"threads_started":1}
I20260926 11:34:01.860641 21154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 748d80d4fd2743d9bc55cf385cab3621: Neither blocks nor log segments found. Creating new log.
I20260926 11:34:01.861362 21154 log.cc:824] T 00000000000000000000000000000000 P 748d80d4fd2743d9bc55cf385cab3621: Log is configured to *not* fsync() on all Append() calls
I20260926 11:34:01.863304 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:34:01.863438 21154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 748d80d4fd2743d9bc55cf385cab3621: No bootstrap required, opened a new log
I20260926 11:34:01.867666 21154 raft_consensus.cc:348] T 00000000000000000000000000000000 P 748d80d4fd2743d9bc55cf385cab3621 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748d80d4fd2743d9bc55cf385cab3621" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44399 } }
I20260926 11:34:01.868117 21154 raft_consensus.cc:374] T 00000000000000000000000000000000 P 748d80d4fd2743d9bc55cf385cab3621 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:34:01.868230 21154 raft_consensus.cc:729] T 00000000000000000000000000000000 P 748d80d4fd2743d9bc55cf385cab3621 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 748d80d4fd2743d9bc55cf385cab3621, State: Initialized, Role: FOLLOWER
I20260926 11:34:01.868782 21154 consensus_queue.cc:260] T 00000000000000000000000000000000 P 748d80d4fd2743d9bc55cf385cab3621 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748d80d4fd2743d9bc55cf385cab3621" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44399 } }
I20260926 11:34:01.868954 21154 raft_consensus.cc:388] T 00000000000000000000000000000000 P 748d80d4fd2743d9bc55cf385cab3621 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:34:01.869028 21154 raft_consensus.cc:482] T 00000000000000000000000000000000 P 748d80d4fd2743d9bc55cf385cab3621 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:34:01.869134 21154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 748d80d4fd2743d9bc55cf385cab3621 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:34:01.869935 21157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:01.865442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37547 (local address 127.0.0.1:44399)
0926 11:34:01.865575 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:01.865579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:01.865777 (+ 198us) server_negotiation.cc:407] Connection header received
0926 11:34:01.865906 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:01.865908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:01.865946 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:01.866014 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:01.866372 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:01.866891 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:01.867994 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:01.868400 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:01.868502 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:01.869470 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:01.869474 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:01.869476 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:34:01.869489 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:01.869516 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:34:01.869529 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:01.869571 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:01.869740 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 11:34:01.869800 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 11:34:01.870303 21154 raft_consensus.cc:504] T 00000000000000000000000000000000 P 748d80d4fd2743d9bc55cf385cab3621 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748d80d4fd2743d9bc55cf385cab3621" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44399 } }
I20260926 11:34:01.870710 21154 leader_election.cc:302] T 00000000000000000000000000000000 P 748d80d4fd2743d9bc55cf385cab3621 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 748d80d4fd2743d9bc55cf385cab3621; no voters:
I20260926 11:34:01.871048 21154 leader_election.cc:288] T 00000000000000000000000000000000 P 748d80d4fd2743d9bc55cf385cab3621 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:34:01.871181 21164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 748d80d4fd2743d9bc55cf385cab3621 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:34:01.871428 21164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 748d80d4fd2743d9bc55cf385cab3621 [term 1 LEADER]: Becoming Leader. State: Replica: 748d80d4fd2743d9bc55cf385cab3621, State: Running, Role: LEADER
I20260926 11:34:01.871774 21164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 748d80d4fd2743d9bc55cf385cab3621 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748d80d4fd2743d9bc55cf385cab3621" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44399 } }
I20260926 11:34:01.872408 21154 sys_catalog.cc:564] T 00000000000000000000000000000000 P 748d80d4fd2743d9bc55cf385cab3621 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:34:01.873400 21165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 748d80d4fd2743d9bc55cf385cab3621 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "748d80d4fd2743d9bc55cf385cab3621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748d80d4fd2743d9bc55cf385cab3621" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44399 } } }
I20260926 11:34:01.873517 21165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 748d80d4fd2743d9bc55cf385cab3621 [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:01.873754 21165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 748d80d4fd2743d9bc55cf385cab3621 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 748d80d4fd2743d9bc55cf385cab3621. Latest consensus state: current_term: 1 leader_uuid: "748d80d4fd2743d9bc55cf385cab3621" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "748d80d4fd2743d9bc55cf385cab3621" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44399 } } }
I20260926 11:34:01.873824 21165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 748d80d4fd2743d9bc55cf385cab3621 [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:01.874110 21173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:34:01.878116 21173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:34:01.886332 21173 catalog_manager.cc:1347] Generated new cluster ID: b11d41caf09c498c85e801d561671fb4
I20260926 11:34:01.886415 21173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:34:01.897457 21173 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:34:01.898388 21173 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:34:01.908023 21173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 748d80d4fd2743d9bc55cf385cab3621: Generated new TSK 0
I20260926 11:34:01.908862 21173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:34:01.923373 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 21086
2026-09-26T11:34:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1443 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:34:01 dist-test-slave-mrck krb5kdc[21185](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:34:01 dist-test-slave-mrck krb5kdc[21185](info): set up 2 sockets
Sep 26 11:34:01 dist-test-slave-mrck krb5kdc[21185](info): commencing operation
krb5kdc: starting...
W20260926 11:34:02.975963 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:34:03 dist-test-slave-mrck krb5kdc[21185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:34:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:34:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 11:34:03.098852 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33819
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:34:03.209019 21198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:34:03.209360 21198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:34:03.209447 21198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:34:03.213563 21198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:34:03.213682 21198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:34:03.213723 21198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:34:03.213763 21198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:34:03.213799 21198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:34:03.219326 21198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:34:03.220542 21198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:34:03.221629 21198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:34:03.230515 21206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:34:03.230520 21204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:34:03.230520 21203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:34:03.230641 21198 server_base.cc:1034] running on GCE node
I20260926 11:34:03.231498 21198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:34:03.232734 21198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:34:03.233897 21198 hybrid_clock.cc:648] HybridClock initialized: now 1790422443233882 us; error 26 us; skew 500 ppm
Sep 26 11:34:03 dist-test-slave-mrck krb5kdc[21185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422443, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:34:03.236819 21198 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 11:34:03.237769 21198 webserver.cc:460] Webserver started at http://127.0.0.1:37039/ using document root <none> and password file <none>
I20260926 11:34:03.238306 21198 fs_manager.cc:359] Metadata directory not provided
I20260926 11:34:03.238375 21198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:34:03.238546 21198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:34:03.240650 21198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "57b723952cd946a6a669f5460637176d"
format_stamp: "Formatted at 2026-09-26 11:34:03 on dist-test-slave-mrck"
server_key: "d5e502261d1bb08c8c59d554db7c2f36"
server_key_iv: "c0882425308c09bd80799e0ed3f809a9"
server_key_version: "encryptionkey@0"
I20260926 11:34:03.241153 21198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "57b723952cd946a6a669f5460637176d"
format_stamp: "Formatted at 2026-09-26 11:34:03 on dist-test-slave-mrck"
server_key: "d5e502261d1bb08c8c59d554db7c2f36"
server_key_iv: "c0882425308c09bd80799e0ed3f809a9"
server_key_version: "encryptionkey@0"
I20260926 11:34:03.244920 21198 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260926 11:34:03.247380 21213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:34:03.248673 21198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 11:34:03.248818 21198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "57b723952cd946a6a669f5460637176d"
format_stamp: "Formatted at 2026-09-26 11:34:03 on dist-test-slave-mrck"
server_key: "d5e502261d1bb08c8c59d554db7c2f36"
server_key_iv: "c0882425308c09bd80799e0ed3f809a9"
server_key_version: "encryptionkey@0"
I20260926 11:34:03.248941 21198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:34:03.253716 21198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:34:03.256718 21198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:34:03.256915 21198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:34:03.264161 21198 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33819
I20260926 11:34:03.264189 21265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33819 every 8 connection(s)
I20260926 11:34:03.265214 21198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:34:03.268229 21266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:34:03.274071 21266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57b723952cd946a6a669f5460637176d: Bootstrap starting.
I20260926 11:34:03.275226 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 21198
I20260926 11:34:03.275399 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:34:03.275682 392 external_mini_cluster.cc:1428] Setting key ffcf280c37319aa6a673ff7ef156051c
I20260926 11:34:03.277191 21266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57b723952cd946a6a669f5460637176d: Neither blocks nor log segments found. Creating new log.
I20260926 11:34:03.277992 21266 log.cc:824] T 00000000000000000000000000000000 P 57b723952cd946a6a669f5460637176d: Log is configured to *not* fsync() on all Append() calls
I20260926 11:34:03.280326 21266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57b723952cd946a6a669f5460637176d: No bootstrap required, opened a new log
I20260926 11:34:03.281242 21269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:03.276919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34169 (local address 127.0.0.1:33819)
0926 11:34:03.277475 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:03.277491 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:03.277531 (+ 40us) server_negotiation.cc:407] Connection header received
0926 11:34:03.278299 (+ 768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:03.278321 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:03.278740 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:03.279143 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:03.280004 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:03.280008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:03.280021 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:34:03.280032 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:03.280088 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:34:03.280107 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:03.280236 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:03.280320 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 11:34:03.280502 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":168,"threads_started":1}
I20260926 11:34:03.284308 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:34:03.284803 21266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57b723952cd946a6a669f5460637176d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b723952cd946a6a669f5460637176d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33819 } }
I20260926 11:34:03.285073 21266 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57b723952cd946a6a669f5460637176d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:34:03.285151 21266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57b723952cd946a6a669f5460637176d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57b723952cd946a6a669f5460637176d, State: Initialized, Role: FOLLOWER
I20260926 11:34:03.285642 21266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57b723952cd946a6a669f5460637176d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b723952cd946a6a669f5460637176d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33819 } }
I20260926 11:34:03.285825 21266 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57b723952cd946a6a669f5460637176d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:34:03.285895 21266 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57b723952cd946a6a669f5460637176d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:34:03.285995 21266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57b723952cd946a6a669f5460637176d [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:34:03.287222 21266 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57b723952cd946a6a669f5460637176d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b723952cd946a6a669f5460637176d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33819 } }
I20260926 11:34:03.287580 21266 leader_election.cc:302] T 00000000000000000000000000000000 P 57b723952cd946a6a669f5460637176d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57b723952cd946a6a669f5460637176d; no voters:
I20260926 11:34:03.287949 21266 leader_election.cc:288] T 00000000000000000000000000000000 P 57b723952cd946a6a669f5460637176d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:34:03.288069 21276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57b723952cd946a6a669f5460637176d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:34:03.288385 21276 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57b723952cd946a6a669f5460637176d [term 1 LEADER]: Becoming Leader. State: Replica: 57b723952cd946a6a669f5460637176d, State: Running, Role: LEADER
I20260926 11:34:03.288779 21276 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57b723952cd946a6a669f5460637176d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b723952cd946a6a669f5460637176d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33819 } }
I20260926 11:34:03.289448 21266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57b723952cd946a6a669f5460637176d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:34:03.289716 21269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:03.286578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49527 (local address 127.0.0.1:33819)
0926 11:34:03.286694 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:03.286698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:03.288038 (+ 1340us) server_negotiation.cc:407] Connection header received
0926 11:34:03.288187 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:03.288192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:03.288237 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:03.288328 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:03.289224 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:03.289228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:03.289253 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:34:03.289265 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:03.289296 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:34:03.289309 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:03.289396 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:03.289560 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 11:34:03.289603 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 11:34:03.290603 21278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57b723952cd946a6a669f5460637176d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57b723952cd946a6a669f5460637176d. Latest consensus state: current_term: 1 leader_uuid: "57b723952cd946a6a669f5460637176d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b723952cd946a6a669f5460637176d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33819 } } }
I20260926 11:34:03.290731 21278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57b723952cd946a6a669f5460637176d [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:03.291038 21277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57b723952cd946a6a669f5460637176d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57b723952cd946a6a669f5460637176d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57b723952cd946a6a669f5460637176d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33819 } } }
I20260926 11:34:03.291122 21285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:34:03.291124 21277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57b723952cd946a6a669f5460637176d [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:03.294503 21285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:34:03.300253 21285 catalog_manager.cc:1347] Generated new cluster ID: 7e3c86927b694d12995f69837ed3e2fc
I20260926 11:34:03.300334 21285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:34:03.321059 21285 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:34:03.321437 21285 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:34:03.328262 21285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57b723952cd946a6a669f5460637176d: Generated new TSK 0
I20260926 11:34:03.329057 21285 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:34:03.338994 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 21198
2026-09-26T11:34:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:34:03 dist-test-slave-mrck krb5kdc[21297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:34:03 dist-test-slave-mrck krb5kdc[21297](info): set up 2 sockets
Sep 26 11:34:03 dist-test-slave-mrck krb5kdc[21297](info): commencing operation
krb5kdc: starting...
W20260926 11:34:04.371920 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:34:04 dist-test-slave-mrck krb5kdc[21297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:34:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:34:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 11:34:04.494658 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41735
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:34:04.605849 21310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:34:04.606154 21310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:34:04.606230 21310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:34:04.610389 21310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:34:04.610463 21310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:34:04.610483 21310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:34:04.610502 21310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:34:04.610517 21310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:34:04.616137 21310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:34:04.617293 21310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:34:04.618376 21310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:34:04.627197 21318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:34:04.627209 21315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:34:04.627194 21316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:34:04.627679 21310 server_base.cc:1034] running on GCE node
I20260926 11:34:04.628355 21310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:34:04.629562 21310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:34:04.630777 21310 hybrid_clock.cc:648] HybridClock initialized: now 1790422444630760 us; error 32 us; skew 500 ppm
Sep 26 11:34:04 dist-test-slave-mrck krb5kdc[21297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:34:04.633687 21310 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 11:34:04.634635 21310 webserver.cc:460] Webserver started at http://127.0.0.1:46783/ using document root <none> and password file <none>
I20260926 11:34:04.635236 21310 fs_manager.cc:359] Metadata directory not provided
I20260926 11:34:04.635336 21310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:34:04.635581 21310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:34:04.637763 21310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "69156344aae841089df89f17e0e834e5"
format_stamp: "Formatted at 2026-09-26 11:34:04 on dist-test-slave-mrck"
server_key: "91f70c3fe4a541eb170d92beb318fc2f"
server_key_iv: "06b8172e5f984d5c6f3e8d32b2a91efa"
server_key_version: "encryptionkey@0"
I20260926 11:34:04.638343 21310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "69156344aae841089df89f17e0e834e5"
format_stamp: "Formatted at 2026-09-26 11:34:04 on dist-test-slave-mrck"
server_key: "91f70c3fe4a541eb170d92beb318fc2f"
server_key_iv: "06b8172e5f984d5c6f3e8d32b2a91efa"
server_key_version: "encryptionkey@0"
I20260926 11:34:04.643445 21310 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260926 11:34:04.647146 21325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:34:04.648805 21310 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:34:04.649000 21310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "69156344aae841089df89f17e0e834e5"
format_stamp: "Formatted at 2026-09-26 11:34:04 on dist-test-slave-mrck"
server_key: "91f70c3fe4a541eb170d92beb318fc2f"
server_key_iv: "06b8172e5f984d5c6f3e8d32b2a91efa"
server_key_version: "encryptionkey@0"
I20260926 11:34:04.649142 21310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:34:04.669040 21310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:34:04.672663 21310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:34:04.672889 21310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:34:04.679908 21310 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41735
I20260926 11:34:04.679942 21377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41735 every 8 connection(s)
I20260926 11:34:04.680934 21310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:34:04.683647 21378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:34:04.689319 21378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69156344aae841089df89f17e0e834e5: Bootstrap starting.
I20260926 11:34:04.691143 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 21310
I20260926 11:34:04.691306 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:34:04.691623 392 external_mini_cluster.cc:1428] Setting key bbdd2615ce8f6bc13d27b8949932d605
I20260926 11:34:04.691972 21378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69156344aae841089df89f17e0e834e5: Neither blocks nor log segments found. Creating new log.
I20260926 11:34:04.692816 21378 log.cc:824] T 00000000000000000000000000000000 P 69156344aae841089df89f17e0e834e5: Log is configured to *not* fsync() on all Append() calls
I20260926 11:34:04.694780 21378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69156344aae841089df89f17e0e834e5: No bootstrap required, opened a new log
I20260926 11:34:04.698253 21378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69156344aae841089df89f17e0e834e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69156344aae841089df89f17e0e834e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41735 } }
I20260926 11:34:04.698520 21378 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69156344aae841089df89f17e0e834e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:34:04.698598 21378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69156344aae841089df89f17e0e834e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69156344aae841089df89f17e0e834e5, State: Initialized, Role: FOLLOWER
I20260926 11:34:04.699138 21378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69156344aae841089df89f17e0e834e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69156344aae841089df89f17e0e834e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41735 } }
I20260926 11:34:04.699314 21378 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69156344aae841089df89f17e0e834e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:34:04.699412 21378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69156344aae841089df89f17e0e834e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:34:04.699539 21378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69156344aae841089df89f17e0e834e5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:34:04.700691 21378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69156344aae841089df89f17e0e834e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69156344aae841089df89f17e0e834e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41735 } }
Sep 26 11:34:04 dist-test-slave-mrck krb5kdc[21297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 11:34:04.701088 21378 leader_election.cc:302] T 00000000000000000000000000000000 P 69156344aae841089df89f17e0e834e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69156344aae841089df89f17e0e834e5; no voters:
I20260926 11:34:04.701469 21378 leader_election.cc:288] T 00000000000000000000000000000000 P 69156344aae841089df89f17e0e834e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:34:04.701538 21383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69156344aae841089df89f17e0e834e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:34:04.701763 21383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69156344aae841089df89f17e0e834e5 [term 1 LEADER]: Becoming Leader. State: Replica: 69156344aae841089df89f17e0e834e5, State: Running, Role: LEADER
I20260926 11:34:04.702437 21383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69156344aae841089df89f17e0e834e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69156344aae841089df89f17e0e834e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41735 } }
I20260926 11:34:04.702898 21378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69156344aae841089df89f17e0e834e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:34:04.704385 21384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69156344aae841089df89f17e0e834e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69156344aae841089df89f17e0e834e5. Latest consensus state: current_term: 1 leader_uuid: "69156344aae841089df89f17e0e834e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69156344aae841089df89f17e0e834e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41735 } } }
I20260926 11:34:04.704396 21385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69156344aae841089df89f17e0e834e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69156344aae841089df89f17e0e834e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69156344aae841089df89f17e0e834e5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41735 } } }
I20260926 11:34:04.704614 21384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69156344aae841089df89f17e0e834e5 [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:04.704617 21385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69156344aae841089df89f17e0e834e5 [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:04.705019 21395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:34:04.707527 21381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:04.692849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:34937 (local address 127.0.0.1:41735)
0926 11:34:04.693356 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:04.693381 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:04.693426 (+ 45us) server_negotiation.cc:407] Connection header received
0926 11:34:04.694158 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:04.694176 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:04.694527 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:04.694887 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:04.695776 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:04.696820 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:04.697723 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:04.698222 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:04.698404 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:04.701632 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:04.701640 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:04.701661 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:04.701695 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:04.703719 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:04.704434 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:04.704442 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:04.704449 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:04.704591 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:04.705017 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:04.705021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:04.705023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:04.705565 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:04.705798 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:04.706571 (+ 773us) server_negotiation.cc:299] Negotiation successful
0926 11:34:04.706783 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":138,"threads_started":1}
I20260926 11:34:04.708467 21395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:34:04.710551 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:34:04.716805 21395 catalog_manager.cc:1347] Generated new cluster ID: 16e863d2c20c44f7a83e3d78a6e7e77d
I20260926 11:34:04.717028 21395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:34:04.724330 21381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:04.713982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:50201 (local address 127.0.0.1:41735)
0926 11:34:04.714099 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:04.714103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:04.714853 (+ 750us) server_negotiation.cc:407] Connection header received
0926 11:34:04.715035 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:04.715038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:04.715086 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:04.715136 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:04.716124 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:04.716712 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:04.717831 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:04.718153 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:04.718232 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:04.720167 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:04.720171 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:04.720174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:04.720202 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:04.722038 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:04.722643 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:04.722647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:04.722649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:04.722731 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:04.723138 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:04.723144 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:04.723148 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:04.723361 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:04.723485 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:04.724034 (+ 549us) server_negotiation.cc:299] Negotiation successful
0926 11:34:04.724126 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":45}
I20260926 11:34:04.725972 21395 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:34:04.726867 21395 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:34:04.735808 21395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69156344aae841089df89f17e0e834e5: Generated new TSK 0
I20260926 11:34:04.736716 21395 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:34:04.749015 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 21310
W20260926 11:34:04.754451 21295 connection.cc:535] client connection to 127.0.0.1:41735 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41735): Connection reset by peer (error 104)
2026-09-26T11:34:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1409 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:34:04 dist-test-slave-mrck krb5kdc[21409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:34:04 dist-test-slave-mrck krb5kdc[21409](info): set up 2 sockets
Sep 26 11:34:04 dist-test-slave-mrck krb5kdc[21409](info): commencing operation
krb5kdc: starting...
W20260926 11:34:05.772009 392 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:34:05 dist-test-slave-mrck krb5kdc[21409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:34:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:34:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 11:34:05.892108 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45471
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:34:06.003201 21422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:34:06.003518 21422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:34:06.003602 21422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:34:06.007685 21422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:34:06.007804 21422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:34:06.007853 21422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:34:06.007902 21422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:34:06.007943 21422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:34:06.013343 21422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:34:06.014430 21422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:34:06.015467 21422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:34:06.024060 21428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:34:06.024204 21422 server_base.cc:1034] running on GCE node
W20260926 11:34:06.024081 21427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:34:06.024394 21430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:34:06.024986 21422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:34:06.026167 21422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:34:06.027362 21422 hybrid_clock.cc:648] HybridClock initialized: now 1790422446027330 us; error 43 us; skew 500 ppm
Sep 26 11:34:06 dist-test-slave-mrck krb5kdc[21409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:34:06.030529 21422 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 11:34:06.031389 21422 webserver.cc:460] Webserver started at http://127.0.0.1:34793/ using document root <none> and password file <none>
I20260926 11:34:06.032019 21422 fs_manager.cc:359] Metadata directory not provided
I20260926 11:34:06.032112 21422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:34:06.032351 21422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:34:06.034380 21422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "8a4ac674c94b4df2ac847cfd595f7353"
format_stamp: "Formatted at 2026-09-26 11:34:06 on dist-test-slave-mrck"
server_key: "92c7fd3778c6379337cf3d0a7f133e40"
server_key_iv: "b8bc1aaac87d935396f52c3a6f530ca7"
server_key_version: "encryptionkey@0"
I20260926 11:34:06.034948 21422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "8a4ac674c94b4df2ac847cfd595f7353"
format_stamp: "Formatted at 2026-09-26 11:34:06 on dist-test-slave-mrck"
server_key: "92c7fd3778c6379337cf3d0a7f133e40"
server_key_iv: "b8bc1aaac87d935396f52c3a6f530ca7"
server_key_version: "encryptionkey@0"
I20260926 11:34:06.038687 21422 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 11:34:06.041139 21437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:34:06.042351 21422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 11:34:06.042490 21422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "8a4ac674c94b4df2ac847cfd595f7353"
format_stamp: "Formatted at 2026-09-26 11:34:06 on dist-test-slave-mrck"
server_key: "92c7fd3778c6379337cf3d0a7f133e40"
server_key_iv: "b8bc1aaac87d935396f52c3a6f530ca7"
server_key_version: "encryptionkey@0"
I20260926 11:34:06.042611 21422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:34:06.066115 21422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:34:06.069586 21422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:34:06.069808 21422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:34:06.077049 21422 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45471
I20260926 11:34:06.077070 21489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45471 every 8 connection(s)
I20260926 11:34:06.078068 21422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:34:06.078608 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 21422
I20260926 11:34:06.078779 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:34:06.079085 392 external_mini_cluster.cc:1428] Setting key b8edd71d52ec1db91de517205539146a
I20260926 11:34:06.081324 21490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:34:06.086952 21490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a4ac674c94b4df2ac847cfd595f7353: Bootstrap starting.
I20260926 11:34:06.088390 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:06.080556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:37153 (local address 127.0.0.1:45471)
0926 11:34:06.080989 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:06.081004 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:06.081038 (+ 34us) server_negotiation.cc:407] Connection header received
0926 11:34:06.081736 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:06.081753 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:06.082110 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:06.082453 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:06.082866 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:06.083944 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:06.084857 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:06.085363 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:06.085578 (+ 215us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:06.086606 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:06.086611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:06.086623 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:34:06.087041 (+ 418us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:06.087122 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:34:06.087152 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:06.087301 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:06.087487 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 11:34:06.087629 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":293,"thread_start_us":134,"threads_started":1}
I20260926 11:34:06.089395 21490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a4ac674c94b4df2ac847cfd595f7353: Neither blocks nor log segments found. Creating new log.
I20260926 11:34:06.090240 21490 log.cc:824] T 00000000000000000000000000000000 P 8a4ac674c94b4df2ac847cfd595f7353: Log is configured to *not* fsync() on all Append() calls
I20260926 11:34:06.091353 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:34:06.092455 21490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a4ac674c94b4df2ac847cfd595f7353: No bootstrap required, opened a new log
I20260926 11:34:06.095662 21490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a4ac674c94b4df2ac847cfd595f7353 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4ac674c94b4df2ac847cfd595f7353" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45471 } }
I20260926 11:34:06.095944 21490 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a4ac674c94b4df2ac847cfd595f7353 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:34:06.096020 21490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a4ac674c94b4df2ac847cfd595f7353 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a4ac674c94b4df2ac847cfd595f7353, State: Initialized, Role: FOLLOWER
I20260926 11:34:06.096489 21490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a4ac674c94b4df2ac847cfd595f7353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4ac674c94b4df2ac847cfd595f7353" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45471 } }
I20260926 11:34:06.096643 21490 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a4ac674c94b4df2ac847cfd595f7353 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:34:06.096712 21490 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a4ac674c94b4df2ac847cfd595f7353 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:34:06.096832 21490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a4ac674c94b4df2ac847cfd595f7353 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:34:06.097888 21493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:06.093597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:38545 (local address 127.0.0.1:45471)
0926 11:34:06.093714 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:06.093717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:06.093936 (+ 219us) server_negotiation.cc:407] Connection header received
0926 11:34:06.094116 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:06.094119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:06.094174 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:06.094263 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:06.094729 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:06.095213 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:06.096157 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:06.096407 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:06.096521 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:06.097419 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:06.097423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:06.097425 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:34:06.097445 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:06.097468 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:34:06.097478 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:06.097572 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:06.097709 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 11:34:06.097749 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 11:34:06.097895 21490 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a4ac674c94b4df2ac847cfd595f7353 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4ac674c94b4df2ac847cfd595f7353" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45471 } }
I20260926 11:34:06.098306 21490 leader_election.cc:302] T 00000000000000000000000000000000 P 8a4ac674c94b4df2ac847cfd595f7353 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a4ac674c94b4df2ac847cfd595f7353; no voters:
I20260926 11:34:06.098666 21490 leader_election.cc:288] T 00000000000000000000000000000000 P 8a4ac674c94b4df2ac847cfd595f7353 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:34:06.098762 21500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a4ac674c94b4df2ac847cfd595f7353 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:34:06.098985 21500 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a4ac674c94b4df2ac847cfd595f7353 [term 1 LEADER]: Becoming Leader. State: Replica: 8a4ac674c94b4df2ac847cfd595f7353, State: Running, Role: LEADER
I20260926 11:34:06.099392 21500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a4ac674c94b4df2ac847cfd595f7353 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4ac674c94b4df2ac847cfd595f7353" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45471 } }
I20260926 11:34:06.099992 21490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a4ac674c94b4df2ac847cfd595f7353 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:34:06.101076 21501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a4ac674c94b4df2ac847cfd595f7353 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a4ac674c94b4df2ac847cfd595f7353" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4ac674c94b4df2ac847cfd595f7353" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45471 } } }
I20260926 11:34:06.101222 21501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a4ac674c94b4df2ac847cfd595f7353 [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:06.101624 21509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:34:06.101594 21502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a4ac674c94b4df2ac847cfd595f7353 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a4ac674c94b4df2ac847cfd595f7353. Latest consensus state: current_term: 1 leader_uuid: "8a4ac674c94b4df2ac847cfd595f7353" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4ac674c94b4df2ac847cfd595f7353" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45471 } } }
I20260926 11:34:06.101781 21502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a4ac674c94b4df2ac847cfd595f7353 [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:06.104611 21509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:34:06.111097 21509 catalog_manager.cc:1347] Generated new cluster ID: 3c7ef67d41d84ce4b0c0968b6ac0640e
I20260926 11:34:06.111182 21509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:34:06.131582 21509 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:34:06.132629 21509 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:34:06.140719 21509 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a4ac674c94b4df2ac847cfd595f7353: Generated new TSK 0
I20260926 11:34:06.141615 21509 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:34:06.156162 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 21422
W20260926 11:34:06.160876 21407 connection.cc:535] client connection to 127.0.0.1:45471 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45471): Connection reset by peer (error 104)
2026-09-26T11:34:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:34:06 dist-test-slave-mrck krb5kdc[21521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:34:06 dist-test-slave-mrck krb5kdc[21521](info): set up 2 sockets
Sep 26 11:34:06 dist-test-slave-mrck krb5kdc[21521](info): commencing operation
krb5kdc: starting...
W20260926 11:34:07.184096 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:34:07 dist-test-slave-mrck krb5kdc[21521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:34:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:34:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 11:34:07.306304 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37957
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:34:07.415831 21534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:34:07.416097 21534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:34:07.416139 21534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:34:07.419987 21534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:34:07.420039 21534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:34:07.420058 21534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:34:07.420075 21534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:34:07.420090 21534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:34:07.425351 21534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:34:07.426340 21534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:34:07.427284 21534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:34:07.435607 21542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:34:07.435612 21540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:34:07.435827 21534 server_base.cc:1034] running on GCE node
W20260926 11:34:07.435986 21539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:34:07.436566 21534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:34:07.437628 21534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:34:07.438802 21534 hybrid_clock.cc:648] HybridClock initialized: now 1790422447438780 us; error 32 us; skew 500 ppm
Sep 26 11:34:07 dist-test-slave-mrck krb5kdc[21521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:34:07.450590 21534 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 11:34:07.451603 21534 webserver.cc:460] Webserver started at http://127.0.0.1:36537/ using document root <none> and password file <none>
I20260926 11:34:07.452260 21534 fs_manager.cc:359] Metadata directory not provided
I20260926 11:34:07.452349 21534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:34:07.452579 21534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:34:07.454639 21534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "c1dae54e1e164f2882adcd69c56130df"
format_stamp: "Formatted at 2026-09-26 11:34:07 on dist-test-slave-mrck"
server_key: "b1e8294b23c8a598bcc4f5652ab6e152"
server_key_iv: "daa0c7ff19dda16a0bcc642c779af18b"
server_key_version: "encryptionkey@0"
I20260926 11:34:07.455225 21534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "c1dae54e1e164f2882adcd69c56130df"
format_stamp: "Formatted at 2026-09-26 11:34:07 on dist-test-slave-mrck"
server_key: "b1e8294b23c8a598bcc4f5652ab6e152"
server_key_iv: "daa0c7ff19dda16a0bcc642c779af18b"
server_key_version: "encryptionkey@0"
I20260926 11:34:07.458842 21534 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 11:34:07.461215 21549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:34:07.462556 21534 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 11:34:07.462716 21534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "c1dae54e1e164f2882adcd69c56130df"
format_stamp: "Formatted at 2026-09-26 11:34:07 on dist-test-slave-mrck"
server_key: "b1e8294b23c8a598bcc4f5652ab6e152"
server_key_iv: "daa0c7ff19dda16a0bcc642c779af18b"
server_key_version: "encryptionkey@0"
I20260926 11:34:07.462841 21534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:34:07.467511 21534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:34:07.470865 21534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:34:07.471068 21534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:34:07.478232 21534 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37957
I20260926 11:34:07.478255 21601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37957 every 8 connection(s)
I20260926 11:34:07.479245 21534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:34:07.481910 21602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:34:07.482506 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 21534
I20260926 11:34:07.482652 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:34:07.482985 392 external_mini_cluster.cc:1428] Setting key 9bc2036109e28fb296eedf4f009ccb78
I20260926 11:34:07.488323 21605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:07.484424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:56975 (local address 127.0.0.1:37957)
0926 11:34:07.484907 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:07.484922 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:07.484958 (+ 36us) server_negotiation.cc:407] Connection header received
0926 11:34:07.485591 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:07.485608 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:07.485900 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:07.486278 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:07.487101 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:07.487106 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:07.487119 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:34:07.487129 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:07.487183 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:34:07.487201 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:07.487354 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:07.487425 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 11:34:07.487587 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":128,"threads_started":1}
I20260926 11:34:07.488341 21602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1dae54e1e164f2882adcd69c56130df: Bootstrap starting.
I20260926 11:34:07.490656 21602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1dae54e1e164f2882adcd69c56130df: Neither blocks nor log segments found. Creating new log.
I20260926 11:34:07.491032 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:34:07.491446 21602 log.cc:824] T 00000000000000000000000000000000 P c1dae54e1e164f2882adcd69c56130df: Log is configured to *not* fsync() on all Append() calls
I20260926 11:34:07.493572 21602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1dae54e1e164f2882adcd69c56130df: No bootstrap required, opened a new log
I20260926 11:34:07.494884 21605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:07.493262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:40583 (local address 127.0.0.1:37957)
0926 11:34:07.493377 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:07.493381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:07.493538 (+ 157us) server_negotiation.cc:407] Connection header received
0926 11:34:07.493665 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:07.493669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:07.493706 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:07.493781 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:07.494456 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:07.494458 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:07.494460 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:34:07.494471 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:07.494495 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:34:07.494505 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:07.494572 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:07.494683 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 11:34:07.494717 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 11:34:07.497413 21602 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1dae54e1e164f2882adcd69c56130df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1dae54e1e164f2882adcd69c56130df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37957 } }
I20260926 11:34:07.497632 21602 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1dae54e1e164f2882adcd69c56130df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:34:07.497705 21602 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1dae54e1e164f2882adcd69c56130df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1dae54e1e164f2882adcd69c56130df, State: Initialized, Role: FOLLOWER
I20260926 11:34:07.498116 21602 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1dae54e1e164f2882adcd69c56130df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1dae54e1e164f2882adcd69c56130df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37957 } }
I20260926 11:34:07.498265 21602 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1dae54e1e164f2882adcd69c56130df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:34:07.498332 21602 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1dae54e1e164f2882adcd69c56130df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:34:07.498446 21602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1dae54e1e164f2882adcd69c56130df [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:34:07.499361 21602 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1dae54e1e164f2882adcd69c56130df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1dae54e1e164f2882adcd69c56130df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37957 } }
I20260926 11:34:07.499701 21602 leader_election.cc:302] T 00000000000000000000000000000000 P c1dae54e1e164f2882adcd69c56130df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1dae54e1e164f2882adcd69c56130df; no voters:
I20260926 11:34:07.500065 21602 leader_election.cc:288] T 00000000000000000000000000000000 P c1dae54e1e164f2882adcd69c56130df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:34:07.500149 21612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1dae54e1e164f2882adcd69c56130df [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:34:07.500334 21612 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1dae54e1e164f2882adcd69c56130df [term 1 LEADER]: Becoming Leader. State: Replica: c1dae54e1e164f2882adcd69c56130df, State: Running, Role: LEADER
I20260926 11:34:07.500634 21612 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1dae54e1e164f2882adcd69c56130df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1dae54e1e164f2882adcd69c56130df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37957 } }
I20260926 11:34:07.501348 21602 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1dae54e1e164f2882adcd69c56130df [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:34:07.501726 21612 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1dae54e1e164f2882adcd69c56130df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1dae54e1e164f2882adcd69c56130df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1dae54e1e164f2882adcd69c56130df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37957 } } }
I20260926 11:34:07.501914 21612 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1dae54e1e164f2882adcd69c56130df [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:07.502143 21613 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1dae54e1e164f2882adcd69c56130df [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1dae54e1e164f2882adcd69c56130df. Latest consensus state: current_term: 1 leader_uuid: "c1dae54e1e164f2882adcd69c56130df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1dae54e1e164f2882adcd69c56130df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37957 } } }
I20260926 11:34:07.502290 21613 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1dae54e1e164f2882adcd69c56130df [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:07.502388 21617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:34:07.505488 21617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:34:07.511040 21617 catalog_manager.cc:1347] Generated new cluster ID: f55241373df24f3bafdea1fc309594a5
I20260926 11:34:07.511123 21617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:34:07.528930 21617 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:34:07.529294 21617 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:34:07.541936 21617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1dae54e1e164f2882adcd69c56130df: Generated new TSK 0
I20260926 11:34:07.542680 21617 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:34:07.551285 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 21534
2026-09-26T11:34:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1396 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:34:07 dist-test-slave-mrck krb5kdc[21633](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:34:07 dist-test-slave-mrck krb5kdc[21633](info): set up 2 sockets
Sep 26 11:34:07 dist-test-slave-mrck krb5kdc[21633](info): commencing operation
krb5kdc: starting...
W20260926 11:34:08.580099 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:34:08 dist-test-slave-mrck krb5kdc[21633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:34:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:34:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 11:34:08.702427 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41153
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:34:08.820102 21646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:34:08.820423 21646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:34:08.820535 21646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:34:08.824740 21646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:34:08.824837 21646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:34:08.824875 21646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:34:08.824937 21646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:34:08.824976 21646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:34:08.830403 21646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41153
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:34:08.831585 21646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:34:08.832666 21646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:34:08.841746 21652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:34:08.841823 21654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:34:08.841810 21646 server_base.cc:1034] running on GCE node
W20260926 11:34:08.841746 21651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:34:08.842620 21646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:34:08.843906 21646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:34:08.845077 21646 hybrid_clock.cc:648] HybridClock initialized: now 1790422448845046 us; error 41 us; skew 500 ppm
Sep 26 11:34:08 dist-test-slave-mrck krb5kdc[21633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:34:08.847836 21646 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 11:34:08.848729 21646 webserver.cc:460] Webserver started at http://127.0.0.1:39641/ using document root <none> and password file <none>
I20260926 11:34:08.849306 21646 fs_manager.cc:359] Metadata directory not provided
I20260926 11:34:08.849398 21646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:34:08.849641 21646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:34:08.851879 21646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "d13f5c07070743f3995c6d9790eb3190"
format_stamp: "Formatted at 2026-09-26 11:34:08 on dist-test-slave-mrck"
server_key: "83e3366c37017751b95d24dbf1c45638"
server_key_iv: "269b815ae607cc8d73365a0f927ebeef"
server_key_version: "encryptionkey@0"
I20260926 11:34:08.852447 21646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "d13f5c07070743f3995c6d9790eb3190"
format_stamp: "Formatted at 2026-09-26 11:34:08 on dist-test-slave-mrck"
server_key: "83e3366c37017751b95d24dbf1c45638"
server_key_iv: "269b815ae607cc8d73365a0f927ebeef"
server_key_version: "encryptionkey@0"
I20260926 11:34:08.856174 21646 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:34:08.858459 21661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:34:08.859867 21646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:34:08.860008 21646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "d13f5c07070743f3995c6d9790eb3190"
format_stamp: "Formatted at 2026-09-26 11:34:08 on dist-test-slave-mrck"
server_key: "83e3366c37017751b95d24dbf1c45638"
server_key_iv: "269b815ae607cc8d73365a0f927ebeef"
server_key_version: "encryptionkey@0"
I20260926 11:34:08.860121 21646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:34:08.871791 21646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:34:08.874903 21646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:34:08.875141 21646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:34:08.882570 21646 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41153
I20260926 11:34:08.882591 21713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41153 every 8 connection(s)
I20260926 11:34:08.883586 21646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:34:08.886349 21714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:34:08.889328 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 21646
I20260926 11:34:08.889437 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:34:08.889727 392 external_mini_cluster.cc:1428] Setting key a9c91c461d2b5d7b93770ef1dbee7c12
I20260926 11:34:08.892423 21714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d13f5c07070743f3995c6d9790eb3190: Bootstrap starting.
I20260926 11:34:08.894604 21714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d13f5c07070743f3995c6d9790eb3190: Neither blocks nor log segments found. Creating new log.
I20260926 11:34:08.895339 21714 log.cc:824] T 00000000000000000000000000000000 P d13f5c07070743f3995c6d9790eb3190: Log is configured to *not* fsync() on all Append() calls
I20260926 11:34:08.897323 21714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d13f5c07070743f3995c6d9790eb3190: No bootstrap required, opened a new log
Sep 26 11:34:08 dist-test-slave-mrck krb5kdc[21633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 11:34:08.901018 21714 raft_consensus.cc:348] T 00000000000000000000000000000000 P d13f5c07070743f3995c6d9790eb3190 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d13f5c07070743f3995c6d9790eb3190" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41153 } }
I20260926 11:34:08.901289 21714 raft_consensus.cc:374] T 00000000000000000000000000000000 P d13f5c07070743f3995c6d9790eb3190 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:34:08.901353 21714 raft_consensus.cc:729] T 00000000000000000000000000000000 P d13f5c07070743f3995c6d9790eb3190 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13f5c07070743f3995c6d9790eb3190, State: Initialized, Role: FOLLOWER
I20260926 11:34:08.901868 21714 consensus_queue.cc:260] T 00000000000000000000000000000000 P d13f5c07070743f3995c6d9790eb3190 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d13f5c07070743f3995c6d9790eb3190" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41153 } }
I20260926 11:34:08.901993 21714 raft_consensus.cc:388] T 00000000000000000000000000000000 P d13f5c07070743f3995c6d9790eb3190 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:34:08.902078 21714 raft_consensus.cc:482] T 00000000000000000000000000000000 P d13f5c07070743f3995c6d9790eb3190 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:34:08.902202 21714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d13f5c07070743f3995c6d9790eb3190 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:34:08.903386 21714 raft_consensus.cc:504] T 00000000000000000000000000000000 P d13f5c07070743f3995c6d9790eb3190 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d13f5c07070743f3995c6d9790eb3190" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41153 } }
I20260926 11:34:08.903795 21714 leader_election.cc:302] T 00000000000000000000000000000000 P d13f5c07070743f3995c6d9790eb3190 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d13f5c07070743f3995c6d9790eb3190; no voters:
I20260926 11:34:08.904158 21714 leader_election.cc:288] T 00000000000000000000000000000000 P d13f5c07070743f3995c6d9790eb3190 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:34:08.904278 21719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d13f5c07070743f3995c6d9790eb3190 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:34:08.904577 21719 raft_consensus.cc:686] T 00000000000000000000000000000000 P d13f5c07070743f3995c6d9790eb3190 [term 1 LEADER]: Becoming Leader. State: Replica: d13f5c07070743f3995c6d9790eb3190, State: Running, Role: LEADER
I20260926 11:34:08.904935 21719 consensus_queue.cc:237] T 00000000000000000000000000000000 P d13f5c07070743f3995c6d9790eb3190 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d13f5c07070743f3995c6d9790eb3190" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41153 } }
I20260926 11:34:08.905710 21714 sys_catalog.cc:564] T 00000000000000000000000000000000 P d13f5c07070743f3995c6d9790eb3190 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:34:08.905717 21717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:08.891164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:43445 (local address 127.0.0.1:41153)
0926 11:34:08.891644 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:08.891660 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:08.891697 (+ 37us) server_negotiation.cc:407] Connection header received
0926 11:34:08.892399 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:08.892423 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:08.892817 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:08.893159 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:08.893962 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:08.895017 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:08.895974 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:08.896450 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:08.896617 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:08.899562 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:08.899567 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:08.899581 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:08.899609 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:08.901990 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:08.902626 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:08.902633 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:08.902638 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:08.902779 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:08.903102 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:08.903106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:08.903108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:08.903653 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:08.903921 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:08.904238 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 11:34:08.904601 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":138,"threads_started":1}
I20260926 11:34:08.906549 21720 sys_catalog.cc:455] T 00000000000000000000000000000000 P d13f5c07070743f3995c6d9790eb3190 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d13f5c07070743f3995c6d9790eb3190" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d13f5c07070743f3995c6d9790eb3190" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41153 } } }
I20260926 11:34:08.906677 21720 sys_catalog.cc:458] T 00000000000000000000000000000000 P d13f5c07070743f3995c6d9790eb3190 [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:08.907038 21721 sys_catalog.cc:455] T 00000000000000000000000000000000 P d13f5c07070743f3995c6d9790eb3190 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d13f5c07070743f3995c6d9790eb3190. Latest consensus state: current_term: 1 leader_uuid: "d13f5c07070743f3995c6d9790eb3190" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d13f5c07070743f3995c6d9790eb3190" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41153 } } }
I20260926 11:34:08.907187 21721 sys_catalog.cc:458] T 00000000000000000000000000000000 P d13f5c07070743f3995c6d9790eb3190 [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:08.907476 21727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:34:08.909353 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:34:08.910501 21727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:34:08.916883 21727 catalog_manager.cc:1347] Generated new cluster ID: 19146140f07c4acd9e748d96fc2f8400
I20260926 11:34:08.916957 21727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:34:08.928917 21717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:08.912774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:52273 (local address 127.0.0.1:41153)
0926 11:34:08.912886 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:08.912891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:08.913113 (+ 222us) server_negotiation.cc:407] Connection header received
0926 11:34:08.913228 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:08.913232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:08.913283 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:08.913371 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:08.914037 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:08.914556 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:08.915591 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:08.916122 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:08.916243 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:08.924984 (+ 8741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:08.924989 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:08.924992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:08.925030 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:08.926900 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:08.927492 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:08.927496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:08.927499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:08.927593 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:08.927940 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:08.927944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:08.927946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:08.928212 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:08.928368 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:08.928621 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 11:34:08.928729 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":30}
I20260926 11:34:08.934593 21727 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:34:08.936033 21727 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:34:08.954279 21727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d13f5c07070743f3995c6d9790eb3190: Generated new TSK 0
I20260926 11:34:08.955336 21727 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:34:08.965107 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 21646
W20260926 11:34:08.970703 21631 connection.cc:535] client connection to 127.0.0.1:41153 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41153): Connection reset by peer (error 104)
2026-09-26T11:34:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:34:09 dist-test-slave-mrck krb5kdc[21745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:34:09 dist-test-slave-mrck krb5kdc[21745](info): set up 2 sockets
Sep 26 11:34:09 dist-test-slave-mrck krb5kdc[21745](info): commencing operation
krb5kdc: starting...
W20260926 11:34:10.008036 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:34:10 dist-test-slave-mrck krb5kdc[21745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:34:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:34:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 11:34:10.134333 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33535
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:34:10.243626 21758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:34:10.243971 21758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:34:10.244055 21758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:34:10.248076 21758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:34:10.248181 21758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:34:10.248211 21758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:34:10.248239 21758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:34:10.248274 21758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:34:10.253999 21758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33535
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:34:10.255119 21758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:34:10.256147 21758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 11:34:10.264391 21758 server_base.cc:1034] running on GCE node
W20260926 11:34:10.264367 21763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:34:10.264397 21764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:34:10.264573 21766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:34:10.265201 21758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:34:10.266249 21758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:34:10.267406 21758 hybrid_clock.cc:648] HybridClock initialized: now 1790422450267380 us; error 42 us; skew 500 ppm
Sep 26 11:34:10 dist-test-slave-mrck krb5kdc[21745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:34:10.270172 21758 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 11:34:10.271126 21758 webserver.cc:460] Webserver started at http://127.0.0.1:35647/ using document root <none> and password file <none>
I20260926 11:34:10.271698 21758 fs_manager.cc:359] Metadata directory not provided
I20260926 11:34:10.271838 21758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:34:10.272078 21758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:34:10.274088 21758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "3a6feecbe99942eb8d3efcff00355753"
format_stamp: "Formatted at 2026-09-26 11:34:10 on dist-test-slave-mrck"
server_key: "04881c6dc0966977795f0ca253743113"
server_key_iv: "f7c8a6aac767f5572bd104260e4eabef"
server_key_version: "encryptionkey@0"
I20260926 11:34:10.274645 21758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "3a6feecbe99942eb8d3efcff00355753"
format_stamp: "Formatted at 2026-09-26 11:34:10 on dist-test-slave-mrck"
server_key: "04881c6dc0966977795f0ca253743113"
server_key_iv: "f7c8a6aac767f5572bd104260e4eabef"
server_key_version: "encryptionkey@0"
I20260926 11:34:10.278331 21758 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260926 11:34:10.280814 21773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:34:10.281999 21758 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:34:10.282145 21758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "3a6feecbe99942eb8d3efcff00355753"
format_stamp: "Formatted at 2026-09-26 11:34:10 on dist-test-slave-mrck"
server_key: "04881c6dc0966977795f0ca253743113"
server_key_iv: "f7c8a6aac767f5572bd104260e4eabef"
server_key_version: "encryptionkey@0"
I20260926 11:34:10.282269 21758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:34:10.313319 21758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:34:10.316697 21758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:34:10.316921 21758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:34:10.324172 21758 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33535
I20260926 11:34:10.324191 21825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33535 every 8 connection(s)
I20260926 11:34:10.325184 21758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:34:10.328052 21826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:34:10.331197 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 21758
I20260926 11:34:10.331348 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:34:10.331653 392 external_mini_cluster.cc:1428] Setting key 2ea23647eabc435d53752688795e1b39
I20260926 11:34:10.334261 21826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a6feecbe99942eb8d3efcff00355753: Bootstrap starting.
I20260926 11:34:10.336552 21826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a6feecbe99942eb8d3efcff00355753: Neither blocks nor log segments found. Creating new log.
I20260926 11:34:10.337514 21826 log.cc:824] T 00000000000000000000000000000000 P 3a6feecbe99942eb8d3efcff00355753: Log is configured to *not* fsync() on all Append() calls
I20260926 11:34:10.339789 21826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a6feecbe99942eb8d3efcff00355753: No bootstrap required, opened a new log
I20260926 11:34:10.341336 21829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:10.333184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:50121 (local address 127.0.0.1:33535)
0926 11:34:10.333632 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:10.333655 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:10.333691 (+ 36us) server_negotiation.cc:407] Connection header received
0926 11:34:10.334314 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:10.334331 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:10.334734 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:10.335092 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:10.335494 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:10.336549 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:10.337828 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:10.338482 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:10.338609 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:10.339471 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:10.339477 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:10.339490 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:34:10.340015 (+ 525us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:10.340078 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:34:10.340101 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:10.340221 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:10.340414 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 11:34:10.340575 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":141,"threads_started":1}
I20260926 11:34:10.343983 21826 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a6feecbe99942eb8d3efcff00355753 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a6feecbe99942eb8d3efcff00355753" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33535 } }
I20260926 11:34:10.344286 21826 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a6feecbe99942eb8d3efcff00355753 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:34:10.344406 21826 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a6feecbe99942eb8d3efcff00355753 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a6feecbe99942eb8d3efcff00355753, State: Initialized, Role: FOLLOWER
I20260926 11:34:10.344568 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:34:10.344887 21826 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a6feecbe99942eb8d3efcff00355753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a6feecbe99942eb8d3efcff00355753" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33535 } }
I20260926 11:34:10.345067 21826 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a6feecbe99942eb8d3efcff00355753 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:34:10.345142 21826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a6feecbe99942eb8d3efcff00355753 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:34:10.345245 21826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a6feecbe99942eb8d3efcff00355753 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:34:10.346500 21826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a6feecbe99942eb8d3efcff00355753 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a6feecbe99942eb8d3efcff00355753" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33535 } }
I20260926 11:34:10.346896 21826 leader_election.cc:302] T 00000000000000000000000000000000 P 3a6feecbe99942eb8d3efcff00355753 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a6feecbe99942eb8d3efcff00355753; no voters:
I20260926 11:34:10.347292 21826 leader_election.cc:288] T 00000000000000000000000000000000 P 3a6feecbe99942eb8d3efcff00355753 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:34:10.347368 21836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a6feecbe99942eb8d3efcff00355753 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:34:10.347645 21836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a6feecbe99942eb8d3efcff00355753 [term 1 LEADER]: Becoming Leader. State: Replica: 3a6feecbe99942eb8d3efcff00355753, State: Running, Role: LEADER
I20260926 11:34:10.348009 21836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a6feecbe99942eb8d3efcff00355753 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a6feecbe99942eb8d3efcff00355753" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33535 } }
I20260926 11:34:10.348584 21826 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a6feecbe99942eb8d3efcff00355753 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:34:10.349229 21836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a6feecbe99942eb8d3efcff00355753 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a6feecbe99942eb8d3efcff00355753. Latest consensus state: current_term: 1 leader_uuid: "3a6feecbe99942eb8d3efcff00355753" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a6feecbe99942eb8d3efcff00355753" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33535 } } }
I20260926 11:34:10.349928 21836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a6feecbe99942eb8d3efcff00355753 [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:10.349808 21837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a6feecbe99942eb8d3efcff00355753 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a6feecbe99942eb8d3efcff00355753" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a6feecbe99942eb8d3efcff00355753" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33535 } } }
I20260926 11:34:10.350226 21837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a6feecbe99942eb8d3efcff00355753 [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:10.351989 21829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:10.346834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:40061 (local address 127.0.0.1:33535)
0926 11:34:10.346953 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:10.346957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:10.347110 (+ 153us) server_negotiation.cc:407] Connection header received
0926 11:34:10.347313 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:10.347316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:10.347368 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:10.347458 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:10.347870 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:10.348628 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:10.349802 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:10.350185 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:10.350269 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:10.351407 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:10.351411 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:10.351414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:34:10.351427 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:10.351453 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:34:10.351465 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:10.351521 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:10.351808 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 11:34:10.351849 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":48}
W20260926 11:34:10.353515 21851 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3a6feecbe99942eb8d3efcff00355753: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:34:10.353605 21851 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:34:10.353686 21846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:34:10.354393 21846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:34:10.360306 21846 catalog_manager.cc:1347] Generated new cluster ID: 788a3657eefb4ba98a4b83f7eb394afb
I20260926 11:34:10.360394 21846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:34:10.375916 21846 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:34:10.376860 21846 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:34:10.382897 21846 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a6feecbe99942eb8d3efcff00355753: Generated new TSK 0
I20260926 11:34:10.383605 21846 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:34:10.391073 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 21758
W20260926 11:34:10.396569 21743 connection.cc:535] client connection to 127.0.0.1:33535 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33535): Connection reset by peer (error 104)
2026-09-26T11:34:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1424 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:34:10 dist-test-slave-mrck krb5kdc[21857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:34:10 dist-test-slave-mrck krb5kdc[21857](info): set up 2 sockets
Sep 26 11:34:10 dist-test-slave-mrck krb5kdc[21857](info): commencing operation
krb5kdc: starting...
W20260926 11:34:11.427991 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:34:11 dist-test-slave-mrck krb5kdc[21857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:34:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:34:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 11:34:11.547046 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39895
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:34:11.658201 21870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:34:11.658489 21870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:34:11.658571 21870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:34:11.662590 21870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:34:11.662680 21870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:34:11.662719 21870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:34:11.662776 21870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:34:11.662815 21870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:34:11.668108 21870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39895
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422180210734-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:34:11.669143 21870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:34:11.670215 21870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:34:11.678519 21876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:34:11.678565 21878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:34:11.678560 21875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:34:11.678828 21870 server_base.cc:1034] running on GCE node
I20260926 11:34:11.679406 21870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:34:11.680527 21870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:34:11.681715 21870 hybrid_clock.cc:648] HybridClock initialized: now 1790422451681702 us; error 27 us; skew 500 ppm
Sep 26 11:34:11 dist-test-slave-mrck krb5kdc[21857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422451, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:34:11.684585 21870 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 11:34:11.685461 21870 webserver.cc:460] Webserver started at http://127.0.0.1:45859/ using document root <none> and password file <none>
I20260926 11:34:11.686045 21870 fs_manager.cc:359] Metadata directory not provided
I20260926 11:34:11.686138 21870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:34:11.686374 21870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:34:11.688493 21870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "a044a78b50bc4703aa259ba57bf5f1b0"
format_stamp: "Formatted at 2026-09-26 11:34:11 on dist-test-slave-mrck"
server_key: "379ebacc37662f45be83e6a3acebb05b"
server_key_iv: "8750e6152cb1e75b330c782cf9039006"
server_key_version: "encryptionkey@0"
I20260926 11:34:11.689061 21870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "a044a78b50bc4703aa259ba57bf5f1b0"
format_stamp: "Formatted at 2026-09-26 11:34:11 on dist-test-slave-mrck"
server_key: "379ebacc37662f45be83e6a3acebb05b"
server_key_iv: "8750e6152cb1e75b330c782cf9039006"
server_key_version: "encryptionkey@0"
I20260926 11:34:11.692811 21870 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 11:34:11.695191 21885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:34:11.696375 21870 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:34:11.696511 21870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "a044a78b50bc4703aa259ba57bf5f1b0"
format_stamp: "Formatted at 2026-09-26 11:34:11 on dist-test-slave-mrck"
server_key: "379ebacc37662f45be83e6a3acebb05b"
server_key_iv: "8750e6152cb1e75b330c782cf9039006"
server_key_version: "encryptionkey@0"
I20260926 11:34:11.696626 21870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:34:11.701022 21870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:34:11.704491 21870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:34:11.704762 21870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:34:11.712450 21870 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39895
I20260926 11:34:11.712502 21937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39895 every 8 connection(s)
I20260926 11:34:11.713430 21870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:34:11.716277 21938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:34:11.721719 21938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a044a78b50bc4703aa259ba57bf5f1b0: Bootstrap starting.
I20260926 11:34:11.723376 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 21870
I20260926 11:34:11.723511 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:34:11.723845 392 external_mini_cluster.cc:1428] Setting key 1db490e61d4c056f94a9cc8986c19a71
I20260926 11:34:11.724465 21938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a044a78b50bc4703aa259ba57bf5f1b0: Neither blocks nor log segments found. Creating new log.
I20260926 11:34:11.725332 21938 log.cc:824] T 00000000000000000000000000000000 P a044a78b50bc4703aa259ba57bf5f1b0: Log is configured to *not* fsync() on all Append() calls
I20260926 11:34:11.727591 21938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a044a78b50bc4703aa259ba57bf5f1b0: No bootstrap required, opened a new log
I20260926 11:34:11.729053 21941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:11.724938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:37657 (local address 127.0.0.1:39895)
0926 11:34:11.725359 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:11.725383 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:11.725445 (+ 62us) server_negotiation.cc:407] Connection header received
0926 11:34:11.726309 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:11.726330 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:11.726606 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:11.726899 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:11.727798 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:11.727804 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:11.727820 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:34:11.727840 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:11.727893 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:34:11.727914 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:11.728003 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:11.728156 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 11:34:11.728340 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":141,"threads_started":1}
I20260926 11:34:11.731626 21938 raft_consensus.cc:348] T 00000000000000000000000000000000 P a044a78b50bc4703aa259ba57bf5f1b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a044a78b50bc4703aa259ba57bf5f1b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39895 } }
I20260926 11:34:11.731946 21938 raft_consensus.cc:374] T 00000000000000000000000000000000 P a044a78b50bc4703aa259ba57bf5f1b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:34:11.732018 21938 raft_consensus.cc:729] T 00000000000000000000000000000000 P a044a78b50bc4703aa259ba57bf5f1b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a044a78b50bc4703aa259ba57bf5f1b0, State: Initialized, Role: FOLLOWER
I20260926 11:34:11.732112 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:34:11.732506 21938 consensus_queue.cc:260] T 00000000000000000000000000000000 P a044a78b50bc4703aa259ba57bf5f1b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a044a78b50bc4703aa259ba57bf5f1b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39895 } }
I20260926 11:34:11.732659 21938 raft_consensus.cc:388] T 00000000000000000000000000000000 P a044a78b50bc4703aa259ba57bf5f1b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:34:11.732745 21938 raft_consensus.cc:482] T 00000000000000000000000000000000 P a044a78b50bc4703aa259ba57bf5f1b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:34:11.732839 21938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a044a78b50bc4703aa259ba57bf5f1b0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:34:11.733839 21938 raft_consensus.cc:504] T 00000000000000000000000000000000 P a044a78b50bc4703aa259ba57bf5f1b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a044a78b50bc4703aa259ba57bf5f1b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39895 } }
I20260926 11:34:11.734215 21938 leader_election.cc:302] T 00000000000000000000000000000000 P a044a78b50bc4703aa259ba57bf5f1b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a044a78b50bc4703aa259ba57bf5f1b0; no voters:
I20260926 11:34:11.734627 21938 leader_election.cc:288] T 00000000000000000000000000000000 P a044a78b50bc4703aa259ba57bf5f1b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:34:11.734750 21948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a044a78b50bc4703aa259ba57bf5f1b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:34:11.734954 21948 raft_consensus.cc:686] T 00000000000000000000000000000000 P a044a78b50bc4703aa259ba57bf5f1b0 [term 1 LEADER]: Becoming Leader. State: Replica: a044a78b50bc4703aa259ba57bf5f1b0, State: Running, Role: LEADER
I20260926 11:34:11.735556 21948 consensus_queue.cc:237] T 00000000000000000000000000000000 P a044a78b50bc4703aa259ba57bf5f1b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a044a78b50bc4703aa259ba57bf5f1b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39895 } }
I20260926 11:34:11.736037 21938 sys_catalog.cc:564] T 00000000000000000000000000000000 P a044a78b50bc4703aa259ba57bf5f1b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:34:11.736249 21941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:11.734230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.58:56051 (local address 127.0.0.1:39895)
0926 11:34:11.734366 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:11.734371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:11.734521 (+ 150us) server_negotiation.cc:407] Connection header received
0926 11:34:11.734598 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:11.734601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:11.734639 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:11.734722 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:11.735654 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:11.735657 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:11.735660 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:34:11.735671 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:11.735697 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:34:11.735710 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:11.735816 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:11.736093 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 11:34:11.736137 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 11:34:11.737247 21949 sys_catalog.cc:455] T 00000000000000000000000000000000 P a044a78b50bc4703aa259ba57bf5f1b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a044a78b50bc4703aa259ba57bf5f1b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a044a78b50bc4703aa259ba57bf5f1b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39895 } } }
I20260926 11:34:11.737370 21949 sys_catalog.cc:458] T 00000000000000000000000000000000 P a044a78b50bc4703aa259ba57bf5f1b0 [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:11.737803 21957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:34:11.738027 21950 sys_catalog.cc:455] T 00000000000000000000000000000000 P a044a78b50bc4703aa259ba57bf5f1b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a044a78b50bc4703aa259ba57bf5f1b0. Latest consensus state: current_term: 1 leader_uuid: "a044a78b50bc4703aa259ba57bf5f1b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a044a78b50bc4703aa259ba57bf5f1b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39895 } } }
I20260926 11:34:11.738181 21950 sys_catalog.cc:458] T 00000000000000000000000000000000 P a044a78b50bc4703aa259ba57bf5f1b0 [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:11.740589 21957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:34:11.745924 21957 catalog_manager.cc:1347] Generated new cluster ID: a40606bc5b6c4c5286318cc63438cae3
I20260926 11:34:11.746019 21957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:34:11.755085 21957 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:34:11.755425 21957 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:34:11.765787 21957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a044a78b50bc4703aa259ba57bf5f1b0: Generated new TSK 0
I20260926 11:34:11.766443 21957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:34:11.772773 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 21870
2026-09-26T11:34:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1380 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:34:11 dist-test-slave-mrck krb5kdc[21969](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:34:11 dist-test-slave-mrck krb5kdc[21969](info): set up 2 sockets
Sep 26 11:34:11 dist-test-slave-mrck krb5kdc[21969](info): commencing operation
krb5kdc: starting...
W20260926 11:34:12.816026 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:34:12 dist-test-slave-mrck krb5kdc[21969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:34:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:34:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:34:12.937038 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38875
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:34:13.049337 21982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:34:13.049696 21982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:34:13.049798 21982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:34:13.054106 21982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:34:13.054203 21982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:34:13.054239 21982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:34:13.054279 21982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:34:13.054319 21982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:34:13.059726 21982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38875
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:34:13.060842 21982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:34:13.061992 21982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:34:13.070773 21990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:34:13.070914 21988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:34:13.070756 21987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:34:13.070827 21982 server_base.cc:1034] running on GCE node
I20260926 11:34:13.071713 21982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:34:13.073027 21982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:34:13.074244 21982 hybrid_clock.cc:648] HybridClock initialized: now 1790422453074226 us; error 36 us; skew 500 ppm
Sep 26 11:34:13 dist-test-slave-mrck krb5kdc[21969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:34:13.081205 21982 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:34:13.082216 21982 webserver.cc:460] Webserver started at http://127.0.98.62:41113/ using document root <none> and password file <none>
I20260926 11:34:13.082818 21982 fs_manager.cc:359] Metadata directory not provided
I20260926 11:34:13.082914 21982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:34:13.083153 21982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:34:13.085251 21982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "06d212dd84a74b65b9c00a0131a6ea3f"
format_stamp: "Formatted at 2026-09-26 11:34:13 on dist-test-slave-mrck"
server_key: "e56fef25cb77518bf3f00d3f415859be"
server_key_iv: "bec0c1f79ba3b04fa5767780e1baa815"
server_key_version: "encryptionkey@0"
I20260926 11:34:13.085819 21982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "06d212dd84a74b65b9c00a0131a6ea3f"
format_stamp: "Formatted at 2026-09-26 11:34:13 on dist-test-slave-mrck"
server_key: "e56fef25cb77518bf3f00d3f415859be"
server_key_iv: "bec0c1f79ba3b04fa5767780e1baa815"
server_key_version: "encryptionkey@0"
I20260926 11:34:13.089614 21982 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 11:34:13.092139 21997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:34:13.093410 21982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 11:34:13.093550 21982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "06d212dd84a74b65b9c00a0131a6ea3f"
format_stamp: "Formatted at 2026-09-26 11:34:13 on dist-test-slave-mrck"
server_key: "e56fef25cb77518bf3f00d3f415859be"
server_key_iv: "bec0c1f79ba3b04fa5767780e1baa815"
server_key_version: "encryptionkey@0"
I20260926 11:34:13.093674 21982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:34:13.116449 21982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:34:13.119942 21982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:34:13.120162 21982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:34:13.127471 21982 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38875
I20260926 11:34:13.127485 22049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38875 every 8 connection(s)
I20260926 11:34:13.128491 21982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:34:13.131572 22050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:34:13.133430 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 21982
I20260926 11:34:13.133595 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:34:13.133878 392 external_mini_cluster.cc:1428] Setting key cf45c50fe15d7ba1d9da27156b727394
I20260926 11:34:13.138875 22050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06d212dd84a74b65b9c00a0131a6ea3f: Bootstrap starting.
I20260926 11:34:13.141957 22050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06d212dd84a74b65b9c00a0131a6ea3f: Neither blocks nor log segments found. Creating new log.
I20260926 11:34:13.143015 22050 log.cc:824] T 00000000000000000000000000000000 P 06d212dd84a74b65b9c00a0131a6ea3f: Log is configured to *not* fsync() on all Append() calls
Sep 26 11:34:13 dist-test-slave-mrck krb5kdc[21969](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:34:13.145494 22050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06d212dd84a74b65b9c00a0131a6ea3f: No bootstrap required, opened a new log
I20260926 11:34:13.149792 22053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:13.135365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49089 (local address 127.0.98.62:38875)
0926 11:34:13.135788 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:13.135810 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:13.135846 (+ 36us) server_negotiation.cc:407] Connection header received
0926 11:34:13.136458 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:13.136474 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:13.136819 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:13.137095 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:13.138257 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:13.139273 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:13.140650 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:13.141134 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:13.141252 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:13.144564 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:13.144570 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:13.144586 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:13.144620 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:13.146603 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:13.147094 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:13.147100 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:13.147114 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:13.147256 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:13.147563 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:13.147566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:13.147568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:13.148189 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:13.148416 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:13.148618 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 11:34:13.148838 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":123,"threads_started":1}
I20260926 11:34:13.150252 22050 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06d212dd84a74b65b9c00a0131a6ea3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d212dd84a74b65b9c00a0131a6ea3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38875 } }
I20260926 11:34:13.150492 22050 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06d212dd84a74b65b9c00a0131a6ea3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:34:13.150563 22050 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06d212dd84a74b65b9c00a0131a6ea3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06d212dd84a74b65b9c00a0131a6ea3f, State: Initialized, Role: FOLLOWER
I20260926 11:34:13.151027 22050 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06d212dd84a74b65b9c00a0131a6ea3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d212dd84a74b65b9c00a0131a6ea3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38875 } }
I20260926 11:34:13.151180 22050 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06d212dd84a74b65b9c00a0131a6ea3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:34:13.151249 22050 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06d212dd84a74b65b9c00a0131a6ea3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:34:13.151345 22050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06d212dd84a74b65b9c00a0131a6ea3f [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:34:13.152421 22050 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06d212dd84a74b65b9c00a0131a6ea3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d212dd84a74b65b9c00a0131a6ea3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38875 } }
I20260926 11:34:13.152755 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:34:13.152812 22050 leader_election.cc:302] T 00000000000000000000000000000000 P 06d212dd84a74b65b9c00a0131a6ea3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06d212dd84a74b65b9c00a0131a6ea3f; no voters:
I20260926 11:34:13.153128 22050 leader_election.cc:288] T 00000000000000000000000000000000 P 06d212dd84a74b65b9c00a0131a6ea3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:34:13.153226 22055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06d212dd84a74b65b9c00a0131a6ea3f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:34:13.153466 22055 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06d212dd84a74b65b9c00a0131a6ea3f [term 1 LEADER]: Becoming Leader. State: Replica: 06d212dd84a74b65b9c00a0131a6ea3f, State: Running, Role: LEADER
I20260926 11:34:13.153805 22055 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06d212dd84a74b65b9c00a0131a6ea3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d212dd84a74b65b9c00a0131a6ea3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38875 } }
I20260926 11:34:13.154518 22050 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06d212dd84a74b65b9c00a0131a6ea3f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:34:13.155243 22057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d212dd84a74b65b9c00a0131a6ea3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06d212dd84a74b65b9c00a0131a6ea3f. Latest consensus state: current_term: 1 leader_uuid: "06d212dd84a74b65b9c00a0131a6ea3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d212dd84a74b65b9c00a0131a6ea3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38875 } } }
I20260926 11:34:13.155424 22056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d212dd84a74b65b9c00a0131a6ea3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06d212dd84a74b65b9c00a0131a6ea3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d212dd84a74b65b9c00a0131a6ea3f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38875 } } }
I20260926 11:34:13.155535 22056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d212dd84a74b65b9c00a0131a6ea3f [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:13.155445 22057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d212dd84a74b65b9c00a0131a6ea3f [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:13.155905 22069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:34:13.159224 22069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:34:13.165695 22069 catalog_manager.cc:1347] Generated new cluster ID: 3047ba59645d49d28000fec97e763870
I20260926 11:34:13.165762 22069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:34:13.165908 22053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:13.154962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43859 (local address 127.0.98.62:38875)
0926 11:34:13.155065 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:13.155070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:13.156180 (+ 1110us) server_negotiation.cc:407] Connection header received
0926 11:34:13.156367 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:13.156370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:13.156412 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:13.156463 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:13.157820 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:13.158551 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:13.159663 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:13.160037 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:13.160116 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:13.162125 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:13.162129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:13.162131 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:13.162156 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:13.163885 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:13.164442 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:13.164445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:13.164447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:13.164529 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:13.164923 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:13.164926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:13.164927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:13.165127 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:13.165241 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:13.165673 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 11:34:13.165772 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 11:34:13.177091 22069 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:34:13.178136 22069 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:34:13.184075 22069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06d212dd84a74b65b9c00a0131a6ea3f: Generated new TSK 0
I20260926 11:34:13.184731 22069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:34:13.196574 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 21982
W20260926 11:34:13.202536 21967 connection.cc:535] client connection to 127.0.98.62:38875 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:38875): Connection reset by peer (error 104)
2026-09-26T11:34:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1425 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:34:13 dist-test-slave-mrck krb5kdc[22081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:34:13 dist-test-slave-mrck krb5kdc[22081](info): set up 2 sockets
Sep 26 11:34:13 dist-test-slave-mrck krb5kdc[22081](info): commencing operation
krb5kdc: starting...
W20260926 11:34:14.223901 392 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:34:14 dist-test-slave-mrck krb5kdc[22081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:34:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:34:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:34:14.348589 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42343
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:34:14.455322 22094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:34:14.455628 22094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:34:14.455704 22094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:34:14.459635 22094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:34:14.459725 22094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:34:14.459774 22094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:34:14.459817 22094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:34:14.459873 22094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:34:14.465144 22094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42343
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:34:14.466189 22094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:34:14.467223 22094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:34:14.476352 22100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:34:14.476485 22099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:34:14.476363 22102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:34:14.476621 22094 server_base.cc:1034] running on GCE node
I20260926 11:34:14.477219 22094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:34:14.478276 22094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:34:14.479436 22094 hybrid_clock.cc:648] HybridClock initialized: now 1790422454479405 us; error 40 us; skew 500 ppm
Sep 26 11:34:14 dist-test-slave-mrck krb5kdc[22081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:34:14.482160 22094 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:34:14.483016 22094 webserver.cc:460] Webserver started at http://127.0.98.62:45645/ using document root <none> and password file <none>
I20260926 11:34:14.483561 22094 fs_manager.cc:359] Metadata directory not provided
I20260926 11:34:14.483619 22094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:34:14.483901 22094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:34:14.485904 22094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "385ea30c229f40829fd697ab6fd841f7"
format_stamp: "Formatted at 2026-09-26 11:34:14 on dist-test-slave-mrck"
server_key: "b62adc41680c3d96dcd926e12807602b"
server_key_iv: "af452afc8ffd2109a10c16210d8dd7db"
server_key_version: "encryptionkey@0"
I20260926 11:34:14.486472 22094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "385ea30c229f40829fd697ab6fd841f7"
format_stamp: "Formatted at 2026-09-26 11:34:14 on dist-test-slave-mrck"
server_key: "b62adc41680c3d96dcd926e12807602b"
server_key_iv: "af452afc8ffd2109a10c16210d8dd7db"
server_key_version: "encryptionkey@0"
I20260926 11:34:14.490093 22094 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 11:34:14.492443 22109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:34:14.493607 22094 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:34:14.493762 22094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "385ea30c229f40829fd697ab6fd841f7"
format_stamp: "Formatted at 2026-09-26 11:34:14 on dist-test-slave-mrck"
server_key: "b62adc41680c3d96dcd926e12807602b"
server_key_iv: "af452afc8ffd2109a10c16210d8dd7db"
server_key_version: "encryptionkey@0"
I20260926 11:34:14.493938 22094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:34:14.507565 22094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:34:14.510272 22094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:34:14.510464 22094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:34:14.517787 22094 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42343
I20260926 11:34:14.517804 22161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42343 every 8 connection(s)
I20260926 11:34:14.518712 22094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:34:14.521435 22162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:34:14.524362 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 22094
I20260926 11:34:14.524498 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:34:14.524773 392 external_mini_cluster.cc:1428] Setting key 9c00f66b422617bcf6f30ccb022d4a01
I20260926 11:34:14.527163 22162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 385ea30c229f40829fd697ab6fd841f7: Bootstrap starting.
I20260926 11:34:14.529589 22162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 385ea30c229f40829fd697ab6fd841f7: Neither blocks nor log segments found. Creating new log.
I20260926 11:34:14.530426 22162 log.cc:824] T 00000000000000000000000000000000 P 385ea30c229f40829fd697ab6fd841f7: Log is configured to *not* fsync() on all Append() calls
I20260926 11:34:14.532608 22162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 385ea30c229f40829fd697ab6fd841f7: No bootstrap required, opened a new log
I20260926 11:34:14.533994 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:14.526070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57551 (local address 127.0.98.62:42343)
0926 11:34:14.526467 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:14.526482 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:14.526515 (+ 33us) server_negotiation.cc:407] Connection header received
0926 11:34:14.527095 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:14.527109 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:14.527501 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:14.527838 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:14.528156 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:14.529119 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:14.530329 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:14.530869 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:14.530996 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:14.531966 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:14.531973 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:14.531988 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:34:14.532480 (+ 492us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:14.532547 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:34:14.532577 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:14.532731 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:14.532938 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 11:34:14.533097 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":135,"threads_started":1}
I20260926 11:34:14.536846 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:34:14.537041 22162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 385ea30c229f40829fd697ab6fd841f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385ea30c229f40829fd697ab6fd841f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } }
I20260926 11:34:14.537261 22162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 385ea30c229f40829fd697ab6fd841f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:34:14.537334 22162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 385ea30c229f40829fd697ab6fd841f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 385ea30c229f40829fd697ab6fd841f7, State: Initialized, Role: FOLLOWER
I20260926 11:34:14.537776 22162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 385ea30c229f40829fd697ab6fd841f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385ea30c229f40829fd697ab6fd841f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } }
I20260926 11:34:14.537902 22162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 385ea30c229f40829fd697ab6fd841f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:34:14.538008 22162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 385ea30c229f40829fd697ab6fd841f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:34:14.538111 22162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 385ea30c229f40829fd697ab6fd841f7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:34:14.539145 22162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 385ea30c229f40829fd697ab6fd841f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385ea30c229f40829fd697ab6fd841f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } }
I20260926 11:34:14.539529 22162 leader_election.cc:302] T 00000000000000000000000000000000 P 385ea30c229f40829fd697ab6fd841f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 385ea30c229f40829fd697ab6fd841f7; no voters:
I20260926 11:34:14.539954 22162 leader_election.cc:288] T 00000000000000000000000000000000 P 385ea30c229f40829fd697ab6fd841f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:34:14.540069 22172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 385ea30c229f40829fd697ab6fd841f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:34:14.540301 22172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 385ea30c229f40829fd697ab6fd841f7 [term 1 LEADER]: Becoming Leader. State: Replica: 385ea30c229f40829fd697ab6fd841f7, State: Running, Role: LEADER
I20260926 11:34:14.540665 22172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 385ea30c229f40829fd697ab6fd841f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385ea30c229f40829fd697ab6fd841f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } }
I20260926 11:34:14.541249 22162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 385ea30c229f40829fd697ab6fd841f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:34:14.542228 22174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 385ea30c229f40829fd697ab6fd841f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "385ea30c229f40829fd697ab6fd841f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385ea30c229f40829fd697ab6fd841f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } } }
I20260926 11:34:14.542364 22174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 385ea30c229f40829fd697ab6fd841f7 [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:14.542971 22174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 385ea30c229f40829fd697ab6fd841f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 385ea30c229f40829fd697ab6fd841f7. Latest consensus state: current_term: 1 leader_uuid: "385ea30c229f40829fd697ab6fd841f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "385ea30c229f40829fd697ab6fd841f7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42343 } } }
I20260926 11:34:14.543054 22174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 385ea30c229f40829fd697ab6fd841f7 [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:14.543903 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:14.538909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41713 (local address 127.0.98.62:42343)
0926 11:34:14.539047 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:14.539053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:14.539170 (+ 117us) server_negotiation.cc:407] Connection header received
0926 11:34:14.539239 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:14.539242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:14.539288 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:14.539370 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:14.539616 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:14.540492 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:14.541586 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:14.541922 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:14.542481 (+ 559us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:14.543057 (+ 576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:14.543063 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:14.543070 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:34:14.543086 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:14.543111 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:34:14.543125 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:14.543214 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:14.543719 (+ 505us) server_negotiation.cc:299] Negotiation successful
0926 11:34:14.543775 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 11:34:14.546012 22187 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 385ea30c229f40829fd697ab6fd841f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:34:14.546257 22187 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:34:14.546356 22183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:34:14.547031 22183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:34:14.552511 22183 catalog_manager.cc:1347] Generated new cluster ID: b4ca06b611884cf48c343856caa2a174
I20260926 11:34:14.552574 22183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:34:14.560359 22183 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:34:14.561308 22183 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:34:14.569726 22183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 385ea30c229f40829fd697ab6fd841f7: Generated new TSK 0
I20260926 11:34:14.570451 22183 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:34:14.577987 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 22094
W20260926 11:34:14.583521 22079 connection.cc:535] client connection to 127.0.98.62:42343 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:42343): Connection reset by peer (error 104)
2026-09-26T11:34:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1380 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:34:14 dist-test-slave-mrck krb5kdc[22193](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:34:14 dist-test-slave-mrck krb5kdc[22193](info): set up 2 sockets
Sep 26 11:34:14 dist-test-slave-mrck krb5kdc[22193](info): commencing operation
krb5kdc: starting...
W20260926 11:34:15.607954 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:34:15 dist-test-slave-mrck krb5kdc[22193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:34:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:34:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:34:15.726265 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37167
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:34:15.838457 22206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:34:15.838766 22206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:34:15.838812 22206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:34:15.842902 22206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:34:15.842978 22206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:34:15.843000 22206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:34:15.843053 22206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:34:15.843089 22206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:34:15.848426 22206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37167
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:34:15.849529 22206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:34:15.850602 22206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:34:15.859141 22211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:34:15.859282 22206 server_base.cc:1034] running on GCE node
W20260926 11:34:15.859141 22212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:34:15.859179 22214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:34:15.860121 22206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:34:15.861163 22206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:34:15.862320 22206 hybrid_clock.cc:648] HybridClock initialized: now 1790422455862278 us; error 54 us; skew 500 ppm
Sep 26 11:34:15 dist-test-slave-mrck krb5kdc[22193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:34:15.865150 22206 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:34:15.866020 22206 webserver.cc:460] Webserver started at http://127.0.98.62:46023/ using document root <none> and password file <none>
I20260926 11:34:15.866582 22206 fs_manager.cc:359] Metadata directory not provided
I20260926 11:34:15.866672 22206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:34:15.866906 22206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:34:15.869086 22206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "1bc46311c4a44285aa040ca02bc497f4"
format_stamp: "Formatted at 2026-09-26 11:34:15 on dist-test-slave-mrck"
server_key: "8c97c92f16b80766d502f94d9933cee4"
server_key_iv: "0c292dc06d3af1063274c894140a2fe0"
server_key_version: "encryptionkey@0"
I20260926 11:34:15.869665 22206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "1bc46311c4a44285aa040ca02bc497f4"
format_stamp: "Formatted at 2026-09-26 11:34:15 on dist-test-slave-mrck"
server_key: "8c97c92f16b80766d502f94d9933cee4"
server_key_iv: "0c292dc06d3af1063274c894140a2fe0"
server_key_version: "encryptionkey@0"
I20260926 11:34:15.873663 22206 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 11:34:15.876304 22221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:34:15.877578 22206 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:34:15.877727 22206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "1bc46311c4a44285aa040ca02bc497f4"
format_stamp: "Formatted at 2026-09-26 11:34:15 on dist-test-slave-mrck"
server_key: "8c97c92f16b80766d502f94d9933cee4"
server_key_iv: "0c292dc06d3af1063274c894140a2fe0"
server_key_version: "encryptionkey@0"
I20260926 11:34:15.877854 22206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:34:15.882820 22206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:34:15.886101 22206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:34:15.886302 22206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:34:15.893432 22206 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37167
I20260926 11:34:15.893456 22273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37167 every 8 connection(s)
I20260926 11:34:15.894361 22206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:34:15.897104 22274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:34:15.902370 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 22206
I20260926 11:34:15.902505 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:34:15.902778 392 external_mini_cluster.cc:1428] Setting key a6bde3053c922d4cff28d367b319e4ce
I20260926 11:34:15.902887 22274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bc46311c4a44285aa040ca02bc497f4: Bootstrap starting.
I20260926 11:34:15.905467 22274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1bc46311c4a44285aa040ca02bc497f4: Neither blocks nor log segments found. Creating new log.
I20260926 11:34:15.906186 22274 log.cc:824] T 00000000000000000000000000000000 P 1bc46311c4a44285aa040ca02bc497f4: Log is configured to *not* fsync() on all Append() calls
I20260926 11:34:15.908043 22277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:15.904196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44477 (local address 127.0.98.62:37167)
0926 11:34:15.904652 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:15.904666 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:15.904701 (+ 35us) server_negotiation.cc:407] Connection header received
0926 11:34:15.905362 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:15.905379 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:15.905709 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:15.906046 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:15.906770 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:15.906775 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:15.906797 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:34:15.906806 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:15.906854 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:34:15.906871 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:15.906986 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:15.907094 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 11:34:15.907257 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":145,"threads_started":1}
I20260926 11:34:15.908282 22274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1bc46311c4a44285aa040ca02bc497f4: No bootstrap required, opened a new log
I20260926 11:34:15.910888 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:34:15.912058 22274 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1bc46311c4a44285aa040ca02bc497f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc46311c4a44285aa040ca02bc497f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37167 } }
I20260926 11:34:15.912346 22274 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1bc46311c4a44285aa040ca02bc497f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:34:15.912442 22274 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1bc46311c4a44285aa040ca02bc497f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bc46311c4a44285aa040ca02bc497f4, State: Initialized, Role: FOLLOWER
I20260926 11:34:15.912935 22274 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1bc46311c4a44285aa040ca02bc497f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc46311c4a44285aa040ca02bc497f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37167 } }
I20260926 11:34:15.913106 22274 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1bc46311c4a44285aa040ca02bc497f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:34:15.913249 22274 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1bc46311c4a44285aa040ca02bc497f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:34:15.913381 22274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1bc46311c4a44285aa040ca02bc497f4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:34:15.914451 22274 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1bc46311c4a44285aa040ca02bc497f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc46311c4a44285aa040ca02bc497f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37167 } }
I20260926 11:34:15.914824 22277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:15.913099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45345 (local address 127.0.98.62:37167)
0926 11:34:15.913227 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:15.913232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:15.913436 (+ 204us) server_negotiation.cc:407] Connection header received
0926 11:34:15.913598 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:15.913601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:15.913635 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:15.913723 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:15.914433 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:15.914436 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:15.914438 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:34:15.914449 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:15.914475 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:34:15.914493 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:15.914583 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:15.914655 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 11:34:15.914695 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 11:34:15.914830 22274 leader_election.cc:302] T 00000000000000000000000000000000 P 1bc46311c4a44285aa040ca02bc497f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1bc46311c4a44285aa040ca02bc497f4; no voters:
I20260926 11:34:15.915261 22274 leader_election.cc:288] T 00000000000000000000000000000000 P 1bc46311c4a44285aa040ca02bc497f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:34:15.915354 22284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1bc46311c4a44285aa040ca02bc497f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:34:15.915548 22284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1bc46311c4a44285aa040ca02bc497f4 [term 1 LEADER]: Becoming Leader. State: Replica: 1bc46311c4a44285aa040ca02bc497f4, State: Running, Role: LEADER
I20260926 11:34:15.915947 22284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1bc46311c4a44285aa040ca02bc497f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc46311c4a44285aa040ca02bc497f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37167 } }
I20260926 11:34:15.916487 22274 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1bc46311c4a44285aa040ca02bc497f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:34:15.917579 22285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bc46311c4a44285aa040ca02bc497f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1bc46311c4a44285aa040ca02bc497f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc46311c4a44285aa040ca02bc497f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37167 } } }
I20260926 11:34:15.917726 22285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bc46311c4a44285aa040ca02bc497f4 [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:15.917991 22286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1bc46311c4a44285aa040ca02bc497f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1bc46311c4a44285aa040ca02bc497f4. Latest consensus state: current_term: 1 leader_uuid: "1bc46311c4a44285aa040ca02bc497f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1bc46311c4a44285aa040ca02bc497f4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37167 } } }
I20260926 11:34:15.918080 22293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:34:15.919792 22286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1bc46311c4a44285aa040ca02bc497f4 [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:15.921605 22293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:34:15.926597 22293 catalog_manager.cc:1347] Generated new cluster ID: da44d9de81584fe2bc40f58cf00055e8
I20260926 11:34:15.926662 22293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:34:15.943591 22293 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:34:15.943981 22293 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:34:15.950960 22293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1bc46311c4a44285aa040ca02bc497f4: Generated new TSK 0
I20260926 11:34:15.951779 22293 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:34:15.958717 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 22206
2026-09-26T11:34:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1380 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:34:16 dist-test-slave-mrck krb5kdc[22305](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:34:16 dist-test-slave-mrck krb5kdc[22305](info): set up 2 sockets
Sep 26 11:34:16 dist-test-slave-mrck krb5kdc[22305](info): commencing operation
krb5kdc: starting...
W20260926 11:34:16.988127 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:34:17 dist-test-slave-mrck krb5kdc[22305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:34:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:34:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:34:17.108510 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34569
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:34:17.217486 22318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:34:17.217785 22318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:34:17.217859 22318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:34:17.221813 22318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:34:17.221911 22318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:34:17.221951 22318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:34:17.222005 22318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:34:17.222043 22318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:34:17.227298 22318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:34:17.228384 22318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:34:17.229401 22318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:34:17.237401 22326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:34:17.237457 22323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:34:17.237560 22318 server_base.cc:1034] running on GCE node
W20260926 11:34:17.237476 22324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:34:17.238255 22318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:34:17.239336 22318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:34:17.240518 22318 hybrid_clock.cc:648] HybridClock initialized: now 1790422457240512 us; error 31 us; skew 500 ppm
Sep 26 11:34:17 dist-test-slave-mrck krb5kdc[22305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:34:17.243207 22318 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:34:17.244118 22318 webserver.cc:460] Webserver started at http://127.0.98.62:37233/ using document root <none> and password file <none>
I20260926 11:34:17.244666 22318 fs_manager.cc:359] Metadata directory not provided
I20260926 11:34:17.244755 22318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:34:17.244984 22318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:34:17.246930 22318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "611bac017e114a699fd7ecd61d02738b"
format_stamp: "Formatted at 2026-09-26 11:34:17 on dist-test-slave-mrck"
server_key: "47403ec056ef8c42a57832d88f4f4e61"
server_key_iv: "0d840c488a42531ee1d542f4424aac68"
server_key_version: "encryptionkey@0"
I20260926 11:34:17.247481 22318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "611bac017e114a699fd7ecd61d02738b"
format_stamp: "Formatted at 2026-09-26 11:34:17 on dist-test-slave-mrck"
server_key: "47403ec056ef8c42a57832d88f4f4e61"
server_key_iv: "0d840c488a42531ee1d542f4424aac68"
server_key_version: "encryptionkey@0"
I20260926 11:34:17.251042 22318 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 11:34:17.253410 22333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:34:17.254597 22318 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 11:34:17.254737 22318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "611bac017e114a699fd7ecd61d02738b"
format_stamp: "Formatted at 2026-09-26 11:34:17 on dist-test-slave-mrck"
server_key: "47403ec056ef8c42a57832d88f4f4e61"
server_key_iv: "0d840c488a42531ee1d542f4424aac68"
server_key_version: "encryptionkey@0"
I20260926 11:34:17.254854 22318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:34:17.279842 22318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:34:17.283236 22318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:34:17.283450 22318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:34:17.290561 22318 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34569
I20260926 11:34:17.290586 22385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34569 every 8 connection(s)
I20260926 11:34:17.291520 22318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:34:17.294238 22386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:34:17.294728 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 22318
I20260926 11:34:17.294860 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:34:17.295140 392 external_mini_cluster.cc:1428] Setting key 6d6a14ea7cc5a6688f5218f2a565644b
I20260926 11:34:17.300732 22386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 611bac017e114a699fd7ecd61d02738b: Bootstrap starting.
I20260926 11:34:17.302834 22386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 611bac017e114a699fd7ecd61d02738b: Neither blocks nor log segments found. Creating new log.
I20260926 11:34:17.303655 22386 log.cc:824] T 00000000000000000000000000000000 P 611bac017e114a699fd7ecd61d02738b: Log is configured to *not* fsync() on all Append() calls
Sep 26 11:34:17 dist-test-slave-mrck krb5kdc[22305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:34:17.305552 22386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 611bac017e114a699fd7ecd61d02738b: No bootstrap required, opened a new log
I20260926 11:34:17.309053 22389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:17.296600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55835 (local address 127.0.98.62:34569)
0926 11:34:17.297015 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:17.297030 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:17.297066 (+ 36us) server_negotiation.cc:407] Connection header received
0926 11:34:17.297736 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:17.297752 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:17.298078 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:17.298421 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:17.299194 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:17.300226 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:17.301137 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:17.301608 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:17.301784 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:17.304624 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:17.304629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:17.304642 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:17.304667 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:17.306342 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:17.306787 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:17.306797 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:17.306801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:17.306909 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:17.307194 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:17.307197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:17.307199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:17.307592 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:17.307814 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:17.308070 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 11:34:17.308344 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":129,"threads_started":1}
I20260926 11:34:17.309087 22386 raft_consensus.cc:348] T 00000000000000000000000000000000 P 611bac017e114a699fd7ecd61d02738b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611bac017e114a699fd7ecd61d02738b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34569 } }
I20260926 11:34:17.309398 22386 raft_consensus.cc:374] T 00000000000000000000000000000000 P 611bac017e114a699fd7ecd61d02738b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:34:17.309485 22386 raft_consensus.cc:729] T 00000000000000000000000000000000 P 611bac017e114a699fd7ecd61d02738b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 611bac017e114a699fd7ecd61d02738b, State: Initialized, Role: FOLLOWER
I20260926 11:34:17.309913 22386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 611bac017e114a699fd7ecd61d02738b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611bac017e114a699fd7ecd61d02738b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34569 } }
I20260926 11:34:17.310063 22386 raft_consensus.cc:388] T 00000000000000000000000000000000 P 611bac017e114a699fd7ecd61d02738b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:34:17.310137 22386 raft_consensus.cc:482] T 00000000000000000000000000000000 P 611bac017e114a699fd7ecd61d02738b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:34:17.310238 22386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 611bac017e114a699fd7ecd61d02738b [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:34:17.311234 22386 raft_consensus.cc:504] T 00000000000000000000000000000000 P 611bac017e114a699fd7ecd61d02738b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611bac017e114a699fd7ecd61d02738b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34569 } }
I20260926 11:34:17.311565 22386 leader_election.cc:302] T 00000000000000000000000000000000 P 611bac017e114a699fd7ecd61d02738b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 611bac017e114a699fd7ecd61d02738b; no voters:
I20260926 11:34:17.311960 22386 leader_election.cc:288] T 00000000000000000000000000000000 P 611bac017e114a699fd7ecd61d02738b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:34:17.312027 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:34:17.312047 22391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 611bac017e114a699fd7ecd61d02738b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:34:17.312222 22391 raft_consensus.cc:686] T 00000000000000000000000000000000 P 611bac017e114a699fd7ecd61d02738b [term 1 LEADER]: Becoming Leader. State: Replica: 611bac017e114a699fd7ecd61d02738b, State: Running, Role: LEADER
I20260926 11:34:17.312589 22391 consensus_queue.cc:237] T 00000000000000000000000000000000 P 611bac017e114a699fd7ecd61d02738b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611bac017e114a699fd7ecd61d02738b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34569 } }
I20260926 11:34:17.313172 22386 sys_catalog.cc:564] T 00000000000000000000000000000000 P 611bac017e114a699fd7ecd61d02738b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:34:17.314247 22392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 611bac017e114a699fd7ecd61d02738b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "611bac017e114a699fd7ecd61d02738b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611bac017e114a699fd7ecd61d02738b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34569 } } }
I20260926 11:34:17.314369 22392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 611bac017e114a699fd7ecd61d02738b [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:17.314558 22393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 611bac017e114a699fd7ecd61d02738b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 611bac017e114a699fd7ecd61d02738b. Latest consensus state: current_term: 1 leader_uuid: "611bac017e114a699fd7ecd61d02738b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "611bac017e114a699fd7ecd61d02738b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34569 } } }
I20260926 11:34:17.314654 22393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 611bac017e114a699fd7ecd61d02738b [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:17.314772 22405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:34:17.318042 22405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:34:17.324029 22389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:17.314212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50893 (local address 127.0.98.62:34569)
0926 11:34:17.314324 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:17.314329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:17.315453 (+ 1124us) server_negotiation.cc:407] Connection header received
0926 11:34:17.315637 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:17.315640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:17.315686 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:17.315811 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:17.316602 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:17.317342 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:17.318448 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:17.318778 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:17.318885 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:17.320625 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:17.320628 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:17.320631 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:17.320658 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:17.322312 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:17.322807 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:17.322810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:17.322812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:17.322890 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:17.323161 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:17.323166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:17.323168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:17.323348 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:17.323492 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:17.323763 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 11:34:17.323875 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 11:34:17.324589 22405 catalog_manager.cc:1347] Generated new cluster ID: c03dabbb52cd4531b71187d4a0bf9d2c
I20260926 11:34:17.324697 22405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:34:17.333377 22405 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:34:17.334267 22405 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:34:17.344364 22405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 611bac017e114a699fd7ecd61d02738b: Generated new TSK 0
I20260926 11:34:17.345125 22405 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:34:17.350831 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 22318
2026-09-26T11:34:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:34:17 dist-test-slave-mrck krb5kdc[22417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:34:17 dist-test-slave-mrck krb5kdc[22417](info): set up 2 sockets
Sep 26 11:34:17 dist-test-slave-mrck krb5kdc[22417](info): commencing operation
krb5kdc: starting...
W20260926 11:34:18.376163 392 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:34:18 dist-test-slave-mrck krb5kdc[22417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:34:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:34:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:34:18.495343 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34921
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34921
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:34:18.603549 22430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:34:18.603942 22430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:34:18.604046 22430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:34:18.608090 22430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:34:18.608188 22430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:34:18.608228 22430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:34:18.608264 22430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:34:18.608299 22430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:34:18.613576 22430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:34:18.614648 22430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:34:18.615659 22430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:34:18.624011 22435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:34:18.624145 22430 server_base.cc:1034] running on GCE node
W20260926 11:34:18.624027 22438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:34:18.624046 22436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:34:18.624922 22430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:34:18.625972 22430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:34:18.627157 22430 hybrid_clock.cc:648] HybridClock initialized: now 1790422458627147 us; error 34 us; skew 500 ppm
Sep 26 11:34:18 dist-test-slave-mrck krb5kdc[22417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:34:18.635532 22430 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:34:18.636528 22430 webserver.cc:460] Webserver started at http://127.0.98.62:34707/ using document root <none> and password file <none>
I20260926 11:34:18.637063 22430 fs_manager.cc:359] Metadata directory not provided
I20260926 11:34:18.637122 22430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:34:18.637287 22430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:34:18.639254 22430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "7ad55cec9b4c42b1a3b2b64c85ccbdf3"
format_stamp: "Formatted at 2026-09-26 11:34:18 on dist-test-slave-mrck"
server_key: "4905efb54ae291d7132993ef24a79fd7"
server_key_iv: "ea8e4760577bf8444996b2fa5afb7f4b"
server_key_version: "encryptionkey@0"
I20260926 11:34:18.639820 22430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "7ad55cec9b4c42b1a3b2b64c85ccbdf3"
format_stamp: "Formatted at 2026-09-26 11:34:18 on dist-test-slave-mrck"
server_key: "4905efb54ae291d7132993ef24a79fd7"
server_key_iv: "ea8e4760577bf8444996b2fa5afb7f4b"
server_key_version: "encryptionkey@0"
I20260926 11:34:18.643311 22430 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 11:34:18.645617 22445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:34:18.646864 22430 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 11:34:18.647008 22430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "7ad55cec9b4c42b1a3b2b64c85ccbdf3"
format_stamp: "Formatted at 2026-09-26 11:34:18 on dist-test-slave-mrck"
server_key: "4905efb54ae291d7132993ef24a79fd7"
server_key_iv: "ea8e4760577bf8444996b2fa5afb7f4b"
server_key_version: "encryptionkey@0"
I20260926 11:34:18.647114 22430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:34:18.660699 22430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:34:18.663518 22430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:34:18.663722 22430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:34:18.670912 22430 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34921
I20260926 11:34:18.670922 22497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34921 every 8 connection(s)
I20260926 11:34:18.672077 22430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:34:18.675068 22498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:34:18.680610 22498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ad55cec9b4c42b1a3b2b64c85ccbdf3: Bootstrap starting.
I20260926 11:34:18.681849 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 22430
I20260926 11:34:18.681974 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:34:18.682268 392 external_mini_cluster.cc:1428] Setting key 632fc59f60c8bbfd3903b9c50e8db5fd
I20260926 11:34:18.683393 22498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ad55cec9b4c42b1a3b2b64c85ccbdf3: Neither blocks nor log segments found. Creating new log.
I20260926 11:34:18.684311 22498 log.cc:824] T 00000000000000000000000000000000 P 7ad55cec9b4c42b1a3b2b64c85ccbdf3: Log is configured to *not* fsync() on all Append() calls
I20260926 11:34:18.686477 22498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ad55cec9b4c42b1a3b2b64c85ccbdf3: No bootstrap required, opened a new log
I20260926 11:34:18.690888 22498 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ad55cec9b4c42b1a3b2b64c85ccbdf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad55cec9b4c42b1a3b2b64c85ccbdf3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34921 } }
I20260926 11:34:18.691175 22498 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ad55cec9b4c42b1a3b2b64c85ccbdf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:34:18.691226 22498 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ad55cec9b4c42b1a3b2b64c85ccbdf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ad55cec9b4c42b1a3b2b64c85ccbdf3, State: Initialized, Role: FOLLOWER
I20260926 11:34:18.691687 22498 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ad55cec9b4c42b1a3b2b64c85ccbdf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad55cec9b4c42b1a3b2b64c85ccbdf3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34921 } }
I20260926 11:34:18.691861 22498 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ad55cec9b4c42b1a3b2b64c85ccbdf3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:34:18.691939 22498 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ad55cec9b4c42b1a3b2b64c85ccbdf3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:34:18.692041 22498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ad55cec9b4c42b1a3b2b64c85ccbdf3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:34:18.692183 22501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:18.683429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46343 (local address 127.0.98.62:34921)
0926 11:34:18.683923 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:18.683942 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:18.683985 (+ 43us) server_negotiation.cc:407] Connection header received
0926 11:34:18.684851 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:18.684871 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:18.685285 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:18.685652 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:18.686069 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:18.687506 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:18.688462 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:18.689200 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:18.689400 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:18.690117 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:18.690125 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:18.690144 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:34:18.690658 (+ 514us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:18.690727 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:34:18.690755 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:18.690846 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:18.690955 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 11:34:18.691162 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":148,"threads_started":1}
I20260926 11:34:18.692998 22498 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ad55cec9b4c42b1a3b2b64c85ccbdf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad55cec9b4c42b1a3b2b64c85ccbdf3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34921 } }
I20260926 11:34:18.693372 22498 leader_election.cc:302] T 00000000000000000000000000000000 P 7ad55cec9b4c42b1a3b2b64c85ccbdf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ad55cec9b4c42b1a3b2b64c85ccbdf3; no voters:
I20260926 11:34:18.693692 22498 leader_election.cc:288] T 00000000000000000000000000000000 P 7ad55cec9b4c42b1a3b2b64c85ccbdf3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:34:18.693791 22503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ad55cec9b4c42b1a3b2b64c85ccbdf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:34:18.694080 22503 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ad55cec9b4c42b1a3b2b64c85ccbdf3 [term 1 LEADER]: Becoming Leader. State: Replica: 7ad55cec9b4c42b1a3b2b64c85ccbdf3, State: Running, Role: LEADER
I20260926 11:34:18.694445 22503 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ad55cec9b4c42b1a3b2b64c85ccbdf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad55cec9b4c42b1a3b2b64c85ccbdf3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34921 } }
I20260926 11:34:18.694998 22498 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ad55cec9b4c42b1a3b2b64c85ccbdf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:34:18.695254 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:34:18.696105 22505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad55cec9b4c42b1a3b2b64c85ccbdf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ad55cec9b4c42b1a3b2b64c85ccbdf3. Latest consensus state: current_term: 1 leader_uuid: "7ad55cec9b4c42b1a3b2b64c85ccbdf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad55cec9b4c42b1a3b2b64c85ccbdf3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34921 } } }
I20260926 11:34:18.696237 22505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad55cec9b4c42b1a3b2b64c85ccbdf3 [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:18.696621 22512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:34:18.697432 22504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ad55cec9b4c42b1a3b2b64c85ccbdf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ad55cec9b4c42b1a3b2b64c85ccbdf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ad55cec9b4c42b1a3b2b64c85ccbdf3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34921 } } }
I20260926 11:34:18.697521 22504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ad55cec9b4c42b1a3b2b64c85ccbdf3 [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:18.700276 22512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:34:18.702466 22501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:18.697921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38311 (local address 127.0.98.62:34921)
0926 11:34:18.698036 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:18.698041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:18.698226 (+ 185us) server_negotiation.cc:407] Connection header received
0926 11:34:18.698349 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:18.698352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:18.698415 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:18.698498 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:18.698865 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:18.699654 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:18.700807 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:18.701050 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:18.701170 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:18.702033 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:18.702037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:18.702040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:34:18.702052 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:18.702085 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:34:18.702096 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:18.702172 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:18.702264 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 11:34:18.702313 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 11:34:18.706650 22512 catalog_manager.cc:1347] Generated new cluster ID: 26a90f026bc449d88b0822f632f20758
I20260926 11:34:18.706775 22512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:34:18.727344 22512 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:34:18.728435 22512 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:34:18.737406 22512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ad55cec9b4c42b1a3b2b64c85ccbdf3: Generated new TSK 0
I20260926 11:34:18.738130 22512 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:34:18.744477 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 22430
2026-09-26T11:34:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1391 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:34:18 dist-test-slave-mrck krb5kdc[22529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:34:18 dist-test-slave-mrck krb5kdc[22529](info): set up 2 sockets
Sep 26 11:34:18 dist-test-slave-mrck krb5kdc[22529](info): commencing operation
krb5kdc: starting...
W20260926 11:34:19.772078 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:34:19 dist-test-slave-mrck krb5kdc[22529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:34:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:34:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:34:19.894290 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36177
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:34:20.002928 22542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:34:20.003198 22542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:34:20.003242 22542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:34:20.007210 22542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:34:20.007280 22542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:34:20.007300 22542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:34:20.007318 22542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:34:20.007332 22542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:34:20.012595 22542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:34:20.013594 22542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:34:20.014539 22542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:34:20.023325 22550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:34:20.023334 22548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:34:20.023533 22542 server_base.cc:1034] running on GCE node
W20260926 11:34:20.023347 22547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:34:20.024180 22542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:34:20.025318 22542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:34:20.026484 22542 hybrid_clock.cc:648] HybridClock initialized: now 1790422460026472 us; error 26 us; skew 500 ppm
Sep 26 11:34:20 dist-test-slave-mrck krb5kdc[22529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:34:20.029645 22542 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:34:20.030833 22542 webserver.cc:460] Webserver started at http://127.0.98.62:40059/ using document root <none> and password file <none>
I20260926 11:34:20.031436 22542 fs_manager.cc:359] Metadata directory not provided
I20260926 11:34:20.031498 22542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:34:20.031667 22542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:34:20.033677 22542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "499b7a51e4e342f38aaeff58b8c80536"
format_stamp: "Formatted at 2026-09-26 11:34:20 on dist-test-slave-mrck"
server_key: "78f88690e37a24b54c0365e222e16999"
server_key_iv: "39216362f008dc9a4009600549dfa380"
server_key_version: "encryptionkey@0"
I20260926 11:34:20.034184 22542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "499b7a51e4e342f38aaeff58b8c80536"
format_stamp: "Formatted at 2026-09-26 11:34:20 on dist-test-slave-mrck"
server_key: "78f88690e37a24b54c0365e222e16999"
server_key_iv: "39216362f008dc9a4009600549dfa380"
server_key_version: "encryptionkey@0"
I20260926 11:34:20.037797 22542 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 11:34:20.040256 22557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:34:20.041464 22542 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:34:20.041622 22542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "499b7a51e4e342f38aaeff58b8c80536"
format_stamp: "Formatted at 2026-09-26 11:34:20 on dist-test-slave-mrck"
server_key: "78f88690e37a24b54c0365e222e16999"
server_key_iv: "39216362f008dc9a4009600549dfa380"
server_key_version: "encryptionkey@0"
I20260926 11:34:20.041743 22542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:34:20.046234 22542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:34:20.049117 22542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:34:20.049300 22542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:34:20.056061 22542 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36177
I20260926 11:34:20.056087 22609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36177 every 8 connection(s)
I20260926 11:34:20.056988 22542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:34:20.059625 22610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:34:20.060074 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 22542
I20260926 11:34:20.060202 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:34:20.060482 392 external_mini_cluster.cc:1428] Setting key 52d2acbac9500e9f66294fc808cb43b3
I20260926 11:34:20.065932 22613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:20.061845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47905 (local address 127.0.98.62:36177)
0926 11:34:20.062249 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:20.062268 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:20.062309 (+ 41us) server_negotiation.cc:407] Connection header received
0926 11:34:20.063046 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:20.063068 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:20.063424 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:20.063896 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:20.064570 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:20.064575 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:20.064591 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:34:20.064602 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:20.064666 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:34:20.064694 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:20.064830 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:20.064900 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 11:34:20.065051 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":115,"threads_started":1}
I20260926 11:34:20.066839 22610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 499b7a51e4e342f38aaeff58b8c80536: Bootstrap starting.
I20260926 11:34:20.068636 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:34:20.069234 22610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 499b7a51e4e342f38aaeff58b8c80536: Neither blocks nor log segments found. Creating new log.
I20260926 11:34:20.069998 22610 log.cc:824] T 00000000000000000000000000000000 P 499b7a51e4e342f38aaeff58b8c80536: Log is configured to *not* fsync() on all Append() calls
I20260926 11:34:20.072180 22610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 499b7a51e4e342f38aaeff58b8c80536: No bootstrap required, opened a new log
I20260926 11:34:20.072299 22613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:20.070709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57095 (local address 127.0.98.62:36177)
0926 11:34:20.070812 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:20.070817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:20.070980 (+ 163us) server_negotiation.cc:407] Connection header received
0926 11:34:20.071071 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:20.071073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:20.071105 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:20.071177 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:20.071906 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:20.071910 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:20.071912 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 11:34:20.071924 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:20.071951 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 11:34:20.071965 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:20.072052 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:20.072149 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 11:34:20.072185 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 11:34:20.075646 22610 raft_consensus.cc:348] T 00000000000000000000000000000000 P 499b7a51e4e342f38aaeff58b8c80536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "499b7a51e4e342f38aaeff58b8c80536" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36177 } }
I20260926 11:34:20.075961 22610 raft_consensus.cc:374] T 00000000000000000000000000000000 P 499b7a51e4e342f38aaeff58b8c80536 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 11:34:20.076026 22610 raft_consensus.cc:729] T 00000000000000000000000000000000 P 499b7a51e4e342f38aaeff58b8c80536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 499b7a51e4e342f38aaeff58b8c80536, State: Initialized, Role: FOLLOWER
I20260926 11:34:20.076472 22610 consensus_queue.cc:260] T 00000000000000000000000000000000 P 499b7a51e4e342f38aaeff58b8c80536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "499b7a51e4e342f38aaeff58b8c80536" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36177 } }
I20260926 11:34:20.076618 22610 raft_consensus.cc:388] T 00000000000000000000000000000000 P 499b7a51e4e342f38aaeff58b8c80536 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 11:34:20.076682 22610 raft_consensus.cc:482] T 00000000000000000000000000000000 P 499b7a51e4e342f38aaeff58b8c80536 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 11:34:20.076773 22610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 499b7a51e4e342f38aaeff58b8c80536 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:34:20.077785 22610 raft_consensus.cc:504] T 00000000000000000000000000000000 P 499b7a51e4e342f38aaeff58b8c80536 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "499b7a51e4e342f38aaeff58b8c80536" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36177 } }
I20260926 11:34:20.078135 22610 leader_election.cc:302] T 00000000000000000000000000000000 P 499b7a51e4e342f38aaeff58b8c80536 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 499b7a51e4e342f38aaeff58b8c80536; no voters:
I20260926 11:34:20.078480 22610 leader_election.cc:288] T 00000000000000000000000000000000 P 499b7a51e4e342f38aaeff58b8c80536 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 11:34:20.078562 22620 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 499b7a51e4e342f38aaeff58b8c80536 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:34:20.078750 22620 raft_consensus.cc:686] T 00000000000000000000000000000000 P 499b7a51e4e342f38aaeff58b8c80536 [term 1 LEADER]: Becoming Leader. State: Replica: 499b7a51e4e342f38aaeff58b8c80536, State: Running, Role: LEADER
I20260926 11:34:20.079046 22620 consensus_queue.cc:237] T 00000000000000000000000000000000 P 499b7a51e4e342f38aaeff58b8c80536 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "499b7a51e4e342f38aaeff58b8c80536" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36177 } }
I20260926 11:34:20.079824 22610 sys_catalog.cc:564] T 00000000000000000000000000000000 P 499b7a51e4e342f38aaeff58b8c80536 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:34:20.080860 22622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 499b7a51e4e342f38aaeff58b8c80536 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 499b7a51e4e342f38aaeff58b8c80536. Latest consensus state: current_term: 1 leader_uuid: "499b7a51e4e342f38aaeff58b8c80536" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "499b7a51e4e342f38aaeff58b8c80536" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36177 } } }
I20260926 11:34:20.080880 22621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 499b7a51e4e342f38aaeff58b8c80536 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "499b7a51e4e342f38aaeff58b8c80536" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "499b7a51e4e342f38aaeff58b8c80536" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36177 } } }
I20260926 11:34:20.081032 22622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 499b7a51e4e342f38aaeff58b8c80536 [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:20.081063 22621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 499b7a51e4e342f38aaeff58b8c80536 [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:20.081517 22629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:34:20.084676 22629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:34:20.090301 22629 catalog_manager.cc:1347] Generated new cluster ID: 4962da08314c403b95559ae3edfbf9f1
I20260926 11:34:20.090384 22629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:34:20.099560 22629 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:34:20.099912 22629 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:34:20.109802 22629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 499b7a51e4e342f38aaeff58b8c80536: Generated new TSK 0
I20260926 11:34:20.110526 22629 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 11:34:20.115119 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 22542
2026-09-26T11:34:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1370 ms)
[----------] 18 tests from AuthTokenIssuingTest (25251 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:34:20 dist-test-slave-mrck krb5kdc[22641](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:34:20 dist-test-slave-mrck krb5kdc[22641](info): set up 2 sockets
Sep 26 11:34:20 dist-test-slave-mrck krb5kdc[22641](info): commencing operation
krb5kdc: starting...
W20260926 11:34:21.156072 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:34:21 dist-test-slave-mrck krb5kdc[22641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:34:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:34:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:34:21.272339 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33119
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33119,127.0.98.61:42081,127.0.98.60:32781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:34:21.380882 22654 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 11:34:21.381228 22654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:34:21.381296 22654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:34:21.381376 22654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:34:21.385368 22654 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:34:21.385458 22654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:34:21.385496 22654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:34:21.385533 22654 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:34:21.385568 22654 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:34:21.390818 22654 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33119,127.0.98.61:42081,127.0.98.60:32781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:34:21.391925 22654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:34:21.392958 22654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:34:21.401539 22659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:34:21.401544 22662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:34:21.401638 22660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:34:21.401746 22654 server_base.cc:1034] running on GCE node
I20260926 11:34:21.402410 22654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:34:21.403534 22654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:34:21.404695 22654 hybrid_clock.cc:648] HybridClock initialized: now 1790422461404687 us; error 36 us; skew 500 ppm
Sep 26 11:34:21 dist-test-slave-mrck krb5kdc[22641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:34:21.407445 22654 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:34:21.408453 22654 webserver.cc:460] Webserver started at http://127.0.98.62:34835/ using document root <none> and password file <none>
I20260926 11:34:21.409026 22654 fs_manager.cc:359] Metadata directory not provided
I20260926 11:34:21.409114 22654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:34:21.409327 22654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:34:21.411325 22654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "1557a1f8a10e4adead57ff1979cdc2e4"
format_stamp: "Formatted at 2026-09-26 11:34:21 on dist-test-slave-mrck"
server_key: "53dcf85da206ad0ccc7066dac18605b3"
server_key_iv: "aada65928ba2899a82db58ebcd91b173"
server_key_version: "encryptionkey@0"
I20260926 11:34:21.411943 22654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "1557a1f8a10e4adead57ff1979cdc2e4"
format_stamp: "Formatted at 2026-09-26 11:34:21 on dist-test-slave-mrck"
server_key: "53dcf85da206ad0ccc7066dac18605b3"
server_key_iv: "aada65928ba2899a82db58ebcd91b173"
server_key_version: "encryptionkey@0"
I20260926 11:34:21.415516 22654 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260926 11:34:21.417941 22669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:34:21.419113 22654 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260926 11:34:21.419255 22654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "1557a1f8a10e4adead57ff1979cdc2e4"
format_stamp: "Formatted at 2026-09-26 11:34:21 on dist-test-slave-mrck"
server_key: "53dcf85da206ad0ccc7066dac18605b3"
server_key_iv: "aada65928ba2899a82db58ebcd91b173"
server_key_version: "encryptionkey@0"
I20260926 11:34:21.419368 22654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:34:21.435993 22654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:34:21.438915 22654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:34:21.439129 22654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:34:21.446463 22654 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33119
I20260926 11:34:21.446777 22721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33119 every 8 connection(s)
I20260926 11:34:21.447448 22654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:34:21.448405 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 22654
I20260926 11:34:21.448539 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:34:21.448840 392 external_mini_cluster.cc:1428] Setting key 79f6d277882c8726e65a4cf0ebac2f99
I20260926 11:34:21.450284 22722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:34:21.453908 22722 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33119 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 26 11:34:21 dist-test-slave-mrck krb5kdc[22641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:34:21.469666 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.456301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56142 (local address 127.0.98.62:33119)
0926 11:34:21.456854 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:21.456864 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:21.457043 (+ 179us) server_negotiation.cc:407] Connection header received
0926 11:34:21.458099 (+ 1056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:21.458103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:21.458345 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:21.458553 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:21.459630 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.460550 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.461726 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.462121 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.462305 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:21.464848 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:21.464853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:21.464862 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:21.464881 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:21.466519 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:21.466999 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:21.467004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:21.467006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:21.467110 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:21.467420 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:21.467423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:21.467425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:21.467869 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:21.468015 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:21.468360 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 11:34:21.468607 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"spinlock_wait_cycles":59136,"thread_start_us":153,"threads_started":1}
I20260926 11:34:21.469666 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.456301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33119 (local address 127.0.0.1:56142)
0926 11:34:21.456802 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:21.456853 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 11:34:21.457830 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:34:21.458550 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:34:21.458554 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:34:21.459079 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:34:21.459475 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:21.459493 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.460690 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.460695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:21.461617 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:21.461627 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.462291 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.462296 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:21.462404 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:21.463032 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:34:21.463069 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:34:21.464684 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:34:21.466646 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:21.466658 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:21.466667 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:21.466892 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:21.467213 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:21.467216 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:21.467218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:21.467325 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:21.467985 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:34:21.467991 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:34:21.468133 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 11:34:21.468303 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 11:34:21.468686 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"mutex_wait_us":109,"thread_start_us":152,"threads_started":1}
I20260926 11:34:21.472028 22722 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42081 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:34:21.472817 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.472528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42081 (local address 127.0.0.1:36354)
0926 11:34:21.472679 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:21.472755 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 11:34:21.473038 22670 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42081: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 11:34:21.473999 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.473734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42081 (local address 127.0.0.1:36366)
0926 11:34:21.473870 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:21.473907 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260926 11:34:21.474210 22722 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42081: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42081: connect: Connection refused (error 111)
I20260926 11:34:21.474546 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42081
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33119,127.0.98.61:42081,127.0.98.60:32781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:34:21.523485 22722 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42081 } attempt: 1
I20260926 11:34:21.524384 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.524100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42081 (local address 127.0.0.1:36378)
0926 11:34:21.524238 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:21.524307 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 11:34:21.525121 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.524908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42081 (local address 127.0.0.1:36392)
0926 11:34:21.525015 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:21.525068 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 11:34:21.525283 22722 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42081: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42081: connect: Connection refused (error 111)
W20260926 11:34:21.582368 22729 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 11:34:21.582660 22729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:34:21.582733 22729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:34:21.582808 22729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:34:21.586844 22729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:34:21.586911 22729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:34:21.586930 22729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:34:21.586947 22729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:34:21.586999 22729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:34:21.592310 22729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33119,127.0.98.61:42081,127.0.98.60:32781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:34:21.593379 22729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:34:21.593506 22722 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42081 } attempt: 2
I20260926 11:34:21.594300 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.594002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42081 (local address 127.0.0.1:36396)
0926 11:34:21.594158 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:21.594227 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 11:34:21.594516 22729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 11:34:21.595064 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.594845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42081 (local address 127.0.0.1:36398)
0926 11:34:21.594947 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:21.594992 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42081: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 11:34:21.595201 22722 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42081: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42081: connect: Connection refused (error 111)
W20260926 11:34:21.603103 22735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:34:21.603070 22737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:34:21.603070 22734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:34:21.603410 22729 server_base.cc:1034] running on GCE node
I20260926 11:34:21.604048 22729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:34:21.605194 22729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:34:21.606380 22729 hybrid_clock.cc:648] HybridClock initialized: now 1790422461606366 us; error 30 us; skew 500 ppm
Sep 26 11:34:21 dist-test-slave-mrck krb5kdc[22641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:34:21.609175 22729 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 11:34:21.610028 22729 webserver.cc:460] Webserver started at http://127.0.98.61:37721/ using document root <none> and password file <none>
I20260926 11:34:21.610579 22729 fs_manager.cc:359] Metadata directory not provided
I20260926 11:34:21.610684 22729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:34:21.610908 22729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:34:21.612959 22729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-1/data/instance:
uuid: "45be645b1f704cf38cebf93d90884be6"
format_stamp: "Formatted at 2026-09-26 11:34:21 on dist-test-slave-mrck"
server_key: "ad6e63d3c797e7c81b5661910767238a"
server_key_iv: "73bf35b344fa30a9001c812fac0cc423"
server_key_version: "encryptionkey@0"
I20260926 11:34:21.613520 22729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-1/wal/instance:
uuid: "45be645b1f704cf38cebf93d90884be6"
format_stamp: "Formatted at 2026-09-26 11:34:21 on dist-test-slave-mrck"
server_key: "ad6e63d3c797e7c81b5661910767238a"
server_key_iv: "73bf35b344fa30a9001c812fac0cc423"
server_key_version: "encryptionkey@0"
I20260926 11:34:21.617442 22729 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 11:34:21.619784 22744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:34:21.620986 22729 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:34:21.621125 22729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-1/wal
uuid: "45be645b1f704cf38cebf93d90884be6"
format_stamp: "Formatted at 2026-09-26 11:34:21 on dist-test-slave-mrck"
server_key: "ad6e63d3c797e7c81b5661910767238a"
server_key_iv: "73bf35b344fa30a9001c812fac0cc423"
server_key_version: "encryptionkey@0"
I20260926 11:34:21.621243 22729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:34:21.647599 22729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:34:21.650533 22729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:34:21.650748 22729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:34:21.657882 22729 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42081
I20260926 11:34:21.658162 22796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42081 every 8 connection(s)
I20260926 11:34:21.658885 22729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-1/data/info.pb
I20260926 11:34:21.660560 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 22729
I20260926 11:34:21.660701 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-1/wal/instance
I20260926 11:34:21.660996 392 external_mini_cluster.cc:1428] Setting key 874449f9edbdcde2317c4bbb2d4d09a0
I20260926 11:34:21.661811 22797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:34:21.666206 22797 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33119 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 26 11:34:21 dist-test-slave-mrck krb5kdc[22641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:34:21.681862 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.668079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56156 (local address 127.0.98.62:33119)
0926 11:34:21.668232 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:21.668237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:21.669114 (+ 877us) server_negotiation.cc:407] Connection header received
0926 11:34:21.670209 (+ 1095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:21.670213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:21.670275 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:21.670354 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:21.671906 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.672731 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.674637 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.674900 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.675062 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:21.678012 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:21.678015 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:21.678018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:21.678049 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:21.679459 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:21.680071 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:21.680077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:21.680082 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:21.680187 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:21.680547 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:21.680551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:21.680553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:21.680778 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:21.680885 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:21.681543 (+ 658us) server_negotiation.cc:299] Negotiation successful
0926 11:34:21.681680 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 11:34:21.682739 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.668495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33119 (local address 127.0.0.1:56156)
0926 11:34:21.668951 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:21.668995 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 11:34:21.669848 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:34:21.670555 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:34:21.670562 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:34:21.671158 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:34:21.671696 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:21.671710 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.672879 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.672885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:21.674476 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:21.674500 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.675127 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.675134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:21.675369 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:21.676065 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:34:21.676091 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:34:21.677827 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:34:21.679584 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:21.679590 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:21.679607 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:21.679961 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:21.680300 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:21.680303 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:21.680305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:21.680445 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:21.680884 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:34:21.680891 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:34:21.681154 (+ 263us) client_negotiation.cc:769] Sending connection context
0926 11:34:21.681472 (+ 318us) client_negotiation.cc:240] Negotiation successful
0926 11:34:21.681812 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":108,"threads_started":1}
I20260926 11:34:21.684242 22797 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42081 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 11:34:21.686357 22722 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42081 } attempt: 3
I20260926 11:34:21.686805 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:32781
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33119,127.0.98.61:42081,127.0.98.60:32781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 11:34:21 dist-test-slave-mrck krb5kdc[22641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 11:34:21 dist-test-slave-mrck krb5kdc[22641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 11:34:21.695490 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.684744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42081 (local address 127.0.0.1:36408)
0926 11:34:21.684921 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:21.684932 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 11:34:21.684991 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:34:21.685543 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:34:21.685545 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:34:21.685704 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:34:21.685836 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:21.685845 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.687247 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.687254 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:21.688229 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:21.688242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.688823 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.688826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:21.688910 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:21.689393 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:34:21.689407 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:34:21.690622 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:34:21.693607 (+ 2985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:21.693613 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:21.693615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:21.693859 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:21.694172 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:21.694175 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:21.694177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:21.694254 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:21.694680 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:34:21.694683 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:34:21.694767 (+ 84us) client_negotiation.cc:769] Sending connection context
0926 11:34:21.694885 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 11:34:21.695290 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260926 11:34:21.695504 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.684918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36408 (local address 127.0.98.61:42081)
0926 11:34:21.685177 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:21.685185 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:21.685226 (+ 41us) server_negotiation.cc:407] Connection header received
0926 11:34:21.685265 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:21.685271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:21.685424 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:21.685524 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:21.685997 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.687126 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.688334 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.688715 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.688828 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:21.690751 (+ 1923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:21.690760 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:21.690773 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:21.690815 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:21.693504 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:21.693973 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:21.693984 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:21.693985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:21.694071 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:21.694343 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:21.694346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:21.694348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:21.694574 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:21.694731 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:21.694983 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 11:34:21.695349 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":221,"server-negotiator.queue_time_us":192,"thread_start_us":118,"threads_started":1}
I20260926 11:34:21.697527 22797 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32781 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:34:21.698303 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.686772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42081 (local address 127.0.0.1:36424)
0926 11:34:21.686893 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:21.686910 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 11:34:21.686990 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:34:21.687487 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:34:21.687490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:34:21.687665 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:34:21.687820 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:21.687827 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.688932 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.688936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:21.690458 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:21.690471 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.691155 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.691160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:21.691291 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:21.691694 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:34:21.691707 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:34:21.693072 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:34:21.696610 (+ 3538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:21.696613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:21.696615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:21.696847 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:21.697157 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:21.697160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:21.697162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:21.697259 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:21.697781 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:34:21.697784 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:34:21.697889 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 11:34:21.698007 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 11:34:21.698145 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 11:34:21.698338 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.686968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36424 (local address 127.0.98.61:42081)
0926 11:34:21.687267 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:21.687278 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:21.687293 (+ 15us) server_negotiation.cc:407] Connection header received
0926 11:34:21.687326 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:21.687328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:21.687367 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:21.687471 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:21.687910 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.688779 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.690623 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.691038 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.691164 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:21.693219 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:21.693226 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:21.693230 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:21.693262 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:21.696471 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:21.696952 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:21.696958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:21.696961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:21.697052 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:21.697358 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:21.697364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:21.697367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:21.697620 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:21.697775 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:21.698065 (+ 290us) server_negotiation.cc:299] Negotiation successful
0926 11:34:21.698169 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":224,"thread_start_us":121,"threads_started":1}
I20260926 11:34:21.698366 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.698000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32781 (local address 127.0.0.1:57686)
0926 11:34:21.698185 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:21.698273 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260926 11:34:21.698566 22747 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32781: connect: Connection refused (error 111)
I20260926 11:34:21.699283 22722 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32781 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:34:21.699858 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.699217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32781 (local address 127.0.0.1:57698)
0926 11:34:21.699705 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:21.699792 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":360}
I20260926 11:34:21.699971 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.699693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32781 (local address 127.0.0.1:57706)
0926 11:34:21.699840 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:21.699888 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260926 11:34:21.699980 22797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32781: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32781: connect: Connection refused (error 111)
I20260926 11:34:21.700609 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.700380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32781 (local address 127.0.0.1:57708)
0926 11:34:21.700508 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:21.700552 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 11:34:21.700760 22722 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32781: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32781: connect: Connection refused (error 111)
I20260926 11:34:21.731931 22722 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32781 } attempt: 1
I20260926 11:34:21.732781 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.732487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32781 (local address 127.0.0.1:57718)
0926 11:34:21.732645 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:21.732705 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 11:34:21.733485 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.733281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32781 (local address 127.0.0.1:57720)
0926 11:34:21.733385 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:21.733434 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 11:34:21.733615 22722 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32781: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32781: connect: Connection refused (error 111)
I20260926 11:34:21.749236 22797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32781 } attempt: 1
I20260926 11:34:21.750165 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.749815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32781 (local address 127.0.0.1:57734)
0926 11:34:21.749980 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:21.750061 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 11:34:21.750972 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.750737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32781 (local address 127.0.0.1:57746)
0926 11:34:21.750854 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:21.750909 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 11:34:21.751174 22797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32781: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32781: connect: Connection refused (error 111)
W20260926 11:34:21.800318 22804 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 11:34:21.800593 22804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:34:21.800666 22804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:34:21.800740 22804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:34:21.804908 22804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:34:21.804967 22804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:34:21.804989 22804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:34:21.805034 22804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:34:21.805068 22804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:34:21.808764 22722 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32781 } attempt: 2
I20260926 11:34:21.809556 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.809265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32781 (local address 127.0.0.1:57762)
0926 11:34:21.809402 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:21.809465 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 11:34:21.810302 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.810100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32781 (local address 127.0.0.1:57766)
0926 11:34:21.810193 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:21.810253 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 11:34:21.810431 22722 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32781: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32781: connect: Connection refused (error 111)
I20260926 11:34:21.810822 22804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33119,127.0.98.61:42081,127.0.98.60:32781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:32781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:34:21.811920 22804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:34:21.812913 22804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 11:34:21.819384 22797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32781 } attempt: 2
I20260926 11:34:21.820294 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.819887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32781 (local address 127.0.0.1:57770)
0926 11:34:21.820127 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:21.820194 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260926 11:34:21.821107 22811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:34:21.821069 22810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:34:21.821273 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.821049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32781 (local address 127.0.0.1:57780)
0926 11:34:21.821147 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:21.821197 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 11:34:21.821384 22797 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32781: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32781: connect: Connection refused (error 111)
W20260926 11:34:21.821455 22813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:34:21.821687 22804 server_base.cc:1034] running on GCE node
I20260926 11:34:21.822264 22804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:34:21.823423 22804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:34:21.824597 22804 hybrid_clock.cc:648] HybridClock initialized: now 1790422461824513 us; error 100 us; skew 500 ppm
Sep 26 11:34:21 dist-test-slave-mrck krb5kdc[22641](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:34:21.827383 22804 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 11:34:21.828326 22804 webserver.cc:460] Webserver started at http://127.0.98.60:36593/ using document root <none> and password file <none>
I20260926 11:34:21.829103 22804 fs_manager.cc:359] Metadata directory not provided
I20260926 11:34:21.829186 22804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:34:21.829392 22804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:34:21.831377 22804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-2/data/instance:
uuid: "82785531e6ec47f0a5270860e5cb7ed3"
format_stamp: "Formatted at 2026-09-26 11:34:21 on dist-test-slave-mrck"
server_key: "5aaa7e07372c299eb6ecac7c120f3c31"
server_key_iv: "a52d9f8b419283719a7d4e6060a6a47e"
server_key_version: "encryptionkey@0"
I20260926 11:34:21.831975 22804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-2/wal/instance:
uuid: "82785531e6ec47f0a5270860e5cb7ed3"
format_stamp: "Formatted at 2026-09-26 11:34:21 on dist-test-slave-mrck"
server_key: "5aaa7e07372c299eb6ecac7c120f3c31"
server_key_iv: "a52d9f8b419283719a7d4e6060a6a47e"
server_key_version: "encryptionkey@0"
I20260926 11:34:21.835546 22804 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:34:21.838039 22820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:34:21.839342 22804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 11:34:21.839499 22804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-2/wal
uuid: "82785531e6ec47f0a5270860e5cb7ed3"
format_stamp: "Formatted at 2026-09-26 11:34:21 on dist-test-slave-mrck"
server_key: "5aaa7e07372c299eb6ecac7c120f3c31"
server_key_iv: "a52d9f8b419283719a7d4e6060a6a47e"
server_key_version: "encryptionkey@0"
I20260926 11:34:21.839620 22804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:34:21.864367 22804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:34:21.867959 22804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:34:21.868227 22804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:34:21.876030 22804 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:32781
I20260926 11:34:21.876298 22872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:32781 every 8 connection(s)
I20260926 11:34:21.877174 22804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-2/data/info.pb
I20260926 11:34:21.880087 22873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:34:21.883785 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 22804
I20260926 11:34:21.883934 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790422180210734-392-0/minicluster-data/master-2/wal/instance
I20260926 11:34:21.884249 392 external_mini_cluster.cc:1428] Setting key 7080542d1d0603b49cc686563825161b
I20260926 11:34:21.884053 22873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33119 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 11:34:21 dist-test-slave-mrck krb5kdc[22641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 11:34:21 dist-test-slave-mrck krb5kdc[22641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 11:34:21.897285 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.886377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56162 (local address 127.0.98.62:33119)
0926 11:34:21.886516 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:21.886520 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:21.886788 (+ 268us) server_negotiation.cc:407] Connection header received
0926 11:34:21.887203 (+ 415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:21.887206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:21.887254 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:21.887330 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:21.888345 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.888843 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.890054 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.890384 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.890471 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:21.893206 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:21.893210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:21.893215 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:21.893245 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:21.894955 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:21.895530 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:21.895534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:21.895538 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:21.895628 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:21.896058 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:21.896063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:21.896068 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:21.896337 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:21.896435 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:21.896961 (+ 526us) server_negotiation.cc:299] Negotiation successful
0926 11:34:21.897101 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 11:34:21.898075 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.886264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33119 (local address 127.0.0.1:56162)
0926 11:34:21.886633 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:21.886686 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 11:34:21.886997 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:34:21.887406 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:34:21.887411 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:34:21.887897 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:34:21.888152 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:21.888165 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.888951 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.888955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:21.889830 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:21.889839 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.890520 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.890524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:21.890694 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:21.891363 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:34:21.891378 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:34:21.893049 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:34:21.895071 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:21.895079 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:21.895086 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:21.895431 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:21.895751 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:21.895756 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:21.895757 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:21.895932 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:21.896440 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:34:21.896445 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:34:21.896697 (+ 252us) client_negotiation.cc:769] Sending connection context
0926 11:34:21.896881 (+ 184us) client_negotiation.cc:240] Negotiation successful
0926 11:34:21.897231 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"spinlock_wait_cycles":4352,"thread_start_us":88,"threads_started":1}
I20260926 11:34:21.898088 22876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.885615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57796 (local address 127.0.98.60:32781)
0926 11:34:21.886019 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:21.886033 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:21.886065 (+ 32us) server_negotiation.cc:407] Connection header received
0926 11:34:21.886990 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:21.887013 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:21.887254 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:21.887451 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:21.888280 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.889310 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.890302 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.890762 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.890871 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:21.893707 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:21.893716 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:21.893731 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:21.893756 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:21.895515 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:21.895995 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:21.896000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:21.896001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:21.896102 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:21.896368 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:21.896374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:21.896379 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:21.896681 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:21.896830 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:21.897094 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 11:34:21.897414 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":172,"server-negotiator.queue_time_us":278,"spinlock_wait_cycles":5888,"thread_start_us":132,"threads_started":1}
I20260926 11:34:21.899564 22873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42081 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 11:34:21 dist-test-slave-mrck krb5kdc[22641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 11:34:21 dist-test-slave-mrck krb5kdc[22641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:34:21.908669 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.900091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42081 (local address 127.0.0.1:36432)
0926 11:34:21.900316 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:21.900331 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:34:21.900423 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:34:21.900976 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:34:21.900979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:34:21.901164 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:34:21.901268 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:21.901273 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.902205 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.902208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:21.902712 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:21.902719 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.903180 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.903184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:21.903288 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:21.903774 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:34:21.903795 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:34:21.905005 (+ 1210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:34:21.907187 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:21.907190 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:21.907191 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:21.907374 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:21.907656 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:21.907670 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:21.907672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:21.907786 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:21.908222 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:34:21.908226 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:34:21.908326 (+ 100us) client_negotiation.cc:769] Sending connection context
0926 11:34:21.908425 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 11:34:21.908524 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260926 11:34:21.908782 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.900193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36432 (local address 127.0.98.61:42081)
0926 11:34:21.900303 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:21.900307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:21.900375 (+ 68us) server_negotiation.cc:407] Connection header received
0926 11:34:21.900523 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:21.900527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:21.900578 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:21.900658 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:21.901392 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.902061 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.902825 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.903071 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.903183 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:21.905156 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:21.905159 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:21.905161 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:21.905187 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:21.907072 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:21.907464 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:21.907468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:21.907470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:21.907558 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:21.907869 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:21.907871 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:21.907873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:21.908124 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:21.908215 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:21.908499 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 11:34:21.908616 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 11:34:21.909554 22873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32781 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:34:21.909576 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.900666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56174 (local address 127.0.98.62:33119)
0926 11:34:21.900779 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:21.900784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:21.900798 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:34:21.900930 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:21.900933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:21.900978 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:21.901056 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:21.901709 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.902236 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.902979 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.903197 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.903304 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:21.905754 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:21.905759 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:21.905761 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:21.905791 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:21.907445 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:21.907984 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:21.907987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:21.907988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:21.908068 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:21.908356 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:21.908359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:21.908360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:21.908587 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:21.909214 (+ 627us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:21.909315 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 11:34:21.909430 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 11:34:21.909577 22722 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32781 } attempt: 3
I20260926 11:34:21.913199 22797 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32781 } attempt: 3
Sep 26 11:34:21 dist-test-slave-mrck krb5kdc[22641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 11:34:21 dist-test-slave-mrck krb5kdc[22641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 11:34:21 dist-test-slave-mrck krb5kdc[22641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 11:34:21 dist-test-slave-mrck krb5kdc[22641](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 11:34:21.920258 22877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.910042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32781 (local address 127.0.0.1:57814)
0926 11:34:21.910261 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:21.910275 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:34:21.910348 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:34:21.910774 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:34:21.910777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:34:21.910937 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:34:21.911026 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:21.911034 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.912038 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.912041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:21.912780 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:21.912789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.913372 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.913375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:21.913949 (+ 574us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:21.914406 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:34:21.914416 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:34:21.916373 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:34:21.918585 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:21.918588 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:21.918591 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:21.918849 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:21.919139 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:21.919142 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:21.919143 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:21.919225 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:21.919690 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:34:21.919692 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:34:21.919826 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 11:34:21.919926 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 11:34:21.920073 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260926 11:34:21.920781 22876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.910182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57814 (local address 127.0.98.60:32781)
0926 11:34:21.910495 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:21.910500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:21.910514 (+ 14us) server_negotiation.cc:407] Connection header received
0926 11:34:21.910546 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:21.910548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:21.910598 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:21.910682 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:21.911139 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.911926 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.912895 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.913252 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.913358 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:21.916497 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:21.916500 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:21.916503 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:21.916523 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:21.918465 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:21.918953 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:21.918959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:21.918972 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:21.919048 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:21.919310 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:21.919313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:21.919314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:21.919565 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:21.919656 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:21.920464 (+ 808us) server_negotiation.cc:299] Negotiation successful
0926 11:34:21.920604 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":260,"spinlock_wait_cycles":2816,"thread_start_us":131,"threads_started":1}
I20260926 11:34:21.921211 22725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.909929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32781 (local address 127.0.0.1:57802)
0926 11:34:21.910212 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:21.910225 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 11:34:21.910292 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:34:21.910749 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:34:21.910754 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:34:21.910921 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:34:21.911023 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:21.911030 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.912441 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.912444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:21.913126 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:21.913134 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.913933 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.913936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:21.914014 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:21.914587 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:34:21.914603 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:34:21.916408 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:34:21.919291 (+ 2883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:21.919296 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:21.919300 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:21.919546 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:21.919941 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:21.919944 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:21.919946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:21.920033 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:21.920719 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:34:21.920722 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:34:21.920822 (+ 100us) client_negotiation.cc:769] Sending connection context
0926 11:34:21.920910 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 11:34:21.921059 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219}
I20260926 11:34:21.921571 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.911078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36442 (local address 127.0.98.61:42081)
0926 11:34:21.911181 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:21.911187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:21.911199 (+ 12us) server_negotiation.cc:407] Connection header received
0926 11:34:21.911234 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:21.911239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:21.911282 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:21.911360 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:21.912068 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.912521 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.913623 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.913943 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.915833 (+ 1890us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:21.917334 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:21.917337 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:21.917340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:21.917364 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:21.919139 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:21.919637 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:21.919643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:21.919648 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:21.919726 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:21.920063 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:21.920068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:21.920073 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:21.920299 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:21.921199 (+ 900us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:21.921282 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 11:34:21.921390 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":43}
I20260926 11:34:21.924333 22878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.910156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57802 (local address 127.0.98.60:32781)
0926 11:34:21.910495 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:21.910504 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:21.910515 (+ 11us) server_negotiation.cc:407] Connection header received
0926 11:34:21.910546 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:21.910548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:21.910598 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:21.910720 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:21.911155 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.912317 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.913478 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.913807 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.914188 (+ 381us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:21.916565 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:21.916570 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:21.916572 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:21.916596 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:21.919160 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:21.919716 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:21.919718 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:21.919720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:21.919828 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:21.920148 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:21.920151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:21.920153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:21.920605 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:21.921945 (+ 1340us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:21.922019 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 11:34:21.924149 (+ 2130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":274,"spinlock_wait_cycles":8064}
I20260926 11:34:21.924502 22873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82785531e6ec47f0a5270860e5cb7ed3: Bootstrap starting.
I20260926 11:34:21.925815 22800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.913573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32781 (local address 127.0.0.1:57816)
0926 11:34:21.913719 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:21.913733 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:34:21.913801 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:34:21.914158 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:34:21.914163 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:34:21.914345 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:34:21.914450 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:21.914460 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.915697 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.915703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:21.916615 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:21.916626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.917346 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.917350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:21.917463 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:21.918117 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:34:21.918135 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:34:21.919613 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:34:21.924261 (+ 4648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:21.924264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:21.924266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:21.924500 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:21.924818 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:21.924821 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:21.924823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:21.924906 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:21.925359 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:34:21.925362 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:34:21.925448 (+ 86us) client_negotiation.cc:769] Sending connection context
0926 11:34:21.925522 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 11:34:21.925656 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":23}
I20260926 11:34:21.926348 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.913626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57816 (local address 127.0.98.60:32781)
0926 11:34:21.913909 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:21.913917 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:21.913930 (+ 13us) server_negotiation.cc:407] Connection header received
0926 11:34:21.913962 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:21.913968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:21.914016 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:21.914567 (+ 551us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:21.914681 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.915561 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.916862 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.917234 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.917746 (+ 512us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:21.919775 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:21.919781 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:21.919787 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:21.919831 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:21.924144 (+ 4313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:21.924635 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:21.924638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:21.924640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:21.924718 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:21.925013 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:21.925016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:21.925017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:21.925250 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:21.925992 (+ 742us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:21.926068 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 11:34:21.926186 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"spinlock_wait_cycles":11648,"thread_start_us":90,"threads_started":1}
I20260926 11:34:21.927126 22722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1557a1f8a10e4adead57ff1979cdc2e4: Bootstrap starting.
I20260926 11:34:21.927403 22873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82785531e6ec47f0a5270860e5cb7ed3: Neither blocks nor log segments found. Creating new log.
I20260926 11:34:21.928062 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:34:21.928345 22873 log.cc:824] T 00000000000000000000000000000000 P 82785531e6ec47f0a5270860e5cb7ed3: Log is configured to *not* fsync() on all Append() calls
I20260926 11:34:21.929754 22722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1557a1f8a10e4adead57ff1979cdc2e4: Neither blocks nor log segments found. Creating new log.
I20260926 11:34:21.930471 22873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82785531e6ec47f0a5270860e5cb7ed3: No bootstrap required, opened a new log
I20260926 11:34:21.930570 22722 log.cc:824] T 00000000000000000000000000000000 P 1557a1f8a10e4adead57ff1979cdc2e4: Log is configured to *not* fsync() on all Append() calls
I20260926 11:34:21.929387 22696 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1557a1f8a10e4adead57ff1979cdc2e4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56174
I20260926 11:34:21.930641 22797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45be645b1f704cf38cebf93d90884be6: Bootstrap starting.
I20260926 11:34:21.933030 22797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45be645b1f704cf38cebf93d90884be6: Neither blocks nor log segments found. Creating new log.
I20260926 11:34:21.933121 22722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1557a1f8a10e4adead57ff1979cdc2e4: No bootstrap required, opened a new log
I20260926 11:34:21.933822 22797 log.cc:824] T 00000000000000000000000000000000 P 45be645b1f704cf38cebf93d90884be6: Log is configured to *not* fsync() on all Append() calls
I20260926 11:34:21.934019 22873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82785531e6ec47f0a5270860e5cb7ed3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1557a1f8a10e4adead57ff1979cdc2e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33119 } } peers { permanent_uuid: "45be645b1f704cf38cebf93d90884be6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42081 } } peers { permanent_uuid: "82785531e6ec47f0a5270860e5cb7ed3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32781 } }
I20260926 11:34:21.934239 22873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82785531e6ec47f0a5270860e5cb7ed3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82785531e6ec47f0a5270860e5cb7ed3, State: Initialized, Role: FOLLOWER
I20260926 11:34:21.934567 22873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82785531e6ec47f0a5270860e5cb7ed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1557a1f8a10e4adead57ff1979cdc2e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33119 } } peers { permanent_uuid: "45be645b1f704cf38cebf93d90884be6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42081 } } peers { permanent_uuid: "82785531e6ec47f0a5270860e5cb7ed3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32781 } }
I20260926 11:34:21.935940 22797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45be645b1f704cf38cebf93d90884be6: No bootstrap required, opened a new log
I20260926 11:34:21.935968 22873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82785531e6ec47f0a5270860e5cb7ed3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:34:21.936710 22722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1557a1f8a10e4adead57ff1979cdc2e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1557a1f8a10e4adead57ff1979cdc2e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33119 } } peers { permanent_uuid: "45be645b1f704cf38cebf93d90884be6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42081 } } peers { permanent_uuid: "82785531e6ec47f0a5270860e5cb7ed3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32781 } }
I20260926 11:34:21.936931 22722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1557a1f8a10e4adead57ff1979cdc2e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1557a1f8a10e4adead57ff1979cdc2e4, State: Initialized, Role: FOLLOWER
I20260926 11:34:21.937237 22722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1557a1f8a10e4adead57ff1979cdc2e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1557a1f8a10e4adead57ff1979cdc2e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33119 } } peers { permanent_uuid: "45be645b1f704cf38cebf93d90884be6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42081 } } peers { permanent_uuid: "82785531e6ec47f0a5270860e5cb7ed3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32781 } }
I20260926 11:34:21.937840 22890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1557a1f8a10e4adead57ff1979cdc2e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1557a1f8a10e4adead57ff1979cdc2e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33119 } } peers { permanent_uuid: "45be645b1f704cf38cebf93d90884be6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42081 } } peers { permanent_uuid: "82785531e6ec47f0a5270860e5cb7ed3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32781 } } }
I20260926 11:34:21.938151 22890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1557a1f8a10e4adead57ff1979cdc2e4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:34:21.938640 22722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1557a1f8a10e4adead57ff1979cdc2e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:34:21.938706 22696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1557a1f8a10e4adead57ff1979cdc2e4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 11:34:21.938829 22696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1557a1f8a10e4adead57ff1979cdc2e4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:34:21.940160 22696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1557a1f8a10e4adead57ff1979cdc2e4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1557a1f8a10e4adead57ff1979cdc2e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33119 } } peers { permanent_uuid: "45be645b1f704cf38cebf93d90884be6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42081 } } peers { permanent_uuid: "82785531e6ec47f0a5270860e5cb7ed3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32781 } }
I20260926 11:34:21.940861 22883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82785531e6ec47f0a5270860e5cb7ed3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1557a1f8a10e4adead57ff1979cdc2e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33119 } } peers { permanent_uuid: "45be645b1f704cf38cebf93d90884be6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42081 } } peers { permanent_uuid: "82785531e6ec47f0a5270860e5cb7ed3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32781 } } }
I20260926 11:34:21.940999 22883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82785531e6ec47f0a5270860e5cb7ed3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:34:21.940862 22797 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45be645b1f704cf38cebf93d90884be6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1557a1f8a10e4adead57ff1979cdc2e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33119 } } peers { permanent_uuid: "45be645b1f704cf38cebf93d90884be6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42081 } } peers { permanent_uuid: "82785531e6ec47f0a5270860e5cb7ed3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32781 } }
I20260926 11:34:21.941097 22797 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45be645b1f704cf38cebf93d90884be6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45be645b1f704cf38cebf93d90884be6, State: Initialized, Role: FOLLOWER
I20260926 11:34:21.941409 22797 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45be645b1f704cf38cebf93d90884be6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1557a1f8a10e4adead57ff1979cdc2e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33119 } } peers { permanent_uuid: "45be645b1f704cf38cebf93d90884be6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42081 } } peers { permanent_uuid: "82785531e6ec47f0a5270860e5cb7ed3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32781 } }
I20260926 11:34:21.942036 22907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45be645b1f704cf38cebf93d90884be6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1557a1f8a10e4adead57ff1979cdc2e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33119 } } peers { permanent_uuid: "45be645b1f704cf38cebf93d90884be6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42081 } } peers { permanent_uuid: "82785531e6ec47f0a5270860e5cb7ed3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32781 } } }
I20260926 11:34:21.942216 22907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45be645b1f704cf38cebf93d90884be6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:34:21.942190 22847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1557a1f8a10e4adead57ff1979cdc2e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "82785531e6ec47f0a5270860e5cb7ed3"
I20260926 11:34:21.942395 22847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82785531e6ec47f0a5270860e5cb7ed3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:34:21.942759 22797 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45be645b1f704cf38cebf93d90884be6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:34:21.939787 22804 master_runner.cc:186] Error getting master registration for 127.0.98.62:33119: OK, instance_id { permanent_uuid: "1557a1f8a10e4adead57ff1979cdc2e4" instance_seqno: 1790422461444392 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 11:34:21.943071 22771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1557a1f8a10e4adead57ff1979cdc2e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "45be645b1f704cf38cebf93d90884be6"
I20260926 11:34:21.943156 22804 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 11:34:21.943267 22771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45be645b1f704cf38cebf93d90884be6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:34:21.943715 22847 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 82785531e6ec47f0a5270860e5cb7ed3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1557a1f8a10e4adead57ff1979cdc2e4 in term 1.
I20260926 11:34:21.944729 22771 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 45be645b1f704cf38cebf93d90884be6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1557a1f8a10e4adead57ff1979cdc2e4 in term 1.
I20260926 11:34:21.944816 22672 leader_election.cc:302] T 00000000000000000000000000000000 P 1557a1f8a10e4adead57ff1979cdc2e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1557a1f8a10e4adead57ff1979cdc2e4, 82785531e6ec47f0a5270860e5cb7ed3; no voters:
I20260926 11:34:21.945094 22890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1557a1f8a10e4adead57ff1979cdc2e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:34:21.945463 22890 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1557a1f8a10e4adead57ff1979cdc2e4 [term 1 LEADER]: Becoming Leader. State: Replica: 1557a1f8a10e4adead57ff1979cdc2e4, State: Running, Role: LEADER
I20260926 11:34:21.945801 22890 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1557a1f8a10e4adead57ff1979cdc2e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1557a1f8a10e4adead57ff1979cdc2e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33119 } } peers { permanent_uuid: "45be645b1f704cf38cebf93d90884be6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42081 } } peers { permanent_uuid: "82785531e6ec47f0a5270860e5cb7ed3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32781 } }
W20260926 11:34:21.947019 22895 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 82785531e6ec47f0a5270860e5cb7ed3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:34:21.947101 22895 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:34:21.947970 22911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1557a1f8a10e4adead57ff1979cdc2e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1557a1f8a10e4adead57ff1979cdc2e4. Latest consensus state: current_term: 1 leader_uuid: "1557a1f8a10e4adead57ff1979cdc2e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1557a1f8a10e4adead57ff1979cdc2e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33119 } } peers { permanent_uuid: "45be645b1f704cf38cebf93d90884be6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42081 } } peers { permanent_uuid: "82785531e6ec47f0a5270860e5cb7ed3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32781 } } }
I20260926 11:34:21.948169 22911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1557a1f8a10e4adead57ff1979cdc2e4 [sys.catalog]: This master's current role is: LEADER
W20260926 11:34:21.952091 22906 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1557a1f8a10e4adead57ff1979cdc2e4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:34:21.952170 22906 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:34:21.948637 22654 master_runner.cc:186] Error getting master registration for 127.0.98.61:42081: OK, instance_id { permanent_uuid: "45be645b1f704cf38cebf93d90884be6" instance_seqno: 1790422461655979 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 11:34:21.956120 22654 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 11:34:21.941063 22696 leader_election.cc:288] T 00000000000000000000000000000000 P 1557a1f8a10e4adead57ff1979cdc2e4 [CANDIDATE]: Term 1 election: Requested vote from peers 45be645b1f704cf38cebf93d90884be6 (127.0.98.61:42081), 82785531e6ec47f0a5270860e5cb7ed3 (127.0.98.60:32781)
W20260926 11:34:21.957046 22922 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 45be645b1f704cf38cebf93d90884be6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:34:21.957134 22922 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:34:21.957635 22919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260926 11:34:21.941226 22675 tablet.cc:2369] T 00000000000000000000000000000000 P 1557a1f8a10e4adead57ff1979cdc2e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 11:34:21.958563 22919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 11:34:21.963653 22771 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 45be645b1f704cf38cebf93d90884be6 [term 1 FOLLOWER]: Refusing update from remote peer 1557a1f8a10e4adead57ff1979cdc2e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:34:21.964355 22911 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1557a1f8a10e4adead57ff1979cdc2e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45be645b1f704cf38cebf93d90884be6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42081 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:34:21.964493 22847 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 82785531e6ec47f0a5270860e5cb7ed3 [term 1 FOLLOWER]: Refusing update from remote peer 1557a1f8a10e4adead57ff1979cdc2e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:34:21.965031 22911 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1557a1f8a10e4adead57ff1979cdc2e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82785531e6ec47f0a5270860e5cb7ed3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:34:21.968446 22907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45be645b1f704cf38cebf93d90884be6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1557a1f8a10e4adead57ff1979cdc2e4. Latest consensus state: current_term: 1 leader_uuid: "1557a1f8a10e4adead57ff1979cdc2e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1557a1f8a10e4adead57ff1979cdc2e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33119 } } peers { permanent_uuid: "45be645b1f704cf38cebf93d90884be6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42081 } } peers { permanent_uuid: "82785531e6ec47f0a5270860e5cb7ed3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32781 } } }
I20260926 11:34:21.968590 22907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45be645b1f704cf38cebf93d90884be6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:34:21.969048 22883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82785531e6ec47f0a5270860e5cb7ed3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1557a1f8a10e4adead57ff1979cdc2e4. Latest consensus state: current_term: 1 leader_uuid: "1557a1f8a10e4adead57ff1979cdc2e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1557a1f8a10e4adead57ff1979cdc2e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33119 } } peers { permanent_uuid: "45be645b1f704cf38cebf93d90884be6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42081 } } peers { permanent_uuid: "82785531e6ec47f0a5270860e5cb7ed3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32781 } } }
I20260926 11:34:21.969175 22883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82785531e6ec47f0a5270860e5cb7ed3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:34:21.970860 22890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1557a1f8a10e4adead57ff1979cdc2e4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1557a1f8a10e4adead57ff1979cdc2e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1557a1f8a10e4adead57ff1979cdc2e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33119 } } peers { permanent_uuid: "45be645b1f704cf38cebf93d90884be6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42081 } } peers { permanent_uuid: "82785531e6ec47f0a5270860e5cb7ed3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32781 } } }
I20260926 11:34:21.971002 22890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1557a1f8a10e4adead57ff1979cdc2e4 [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:21.970993 22883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82785531e6ec47f0a5270860e5cb7ed3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1557a1f8a10e4adead57ff1979cdc2e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1557a1f8a10e4adead57ff1979cdc2e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33119 } } peers { permanent_uuid: "45be645b1f704cf38cebf93d90884be6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42081 } } peers { permanent_uuid: "82785531e6ec47f0a5270860e5cb7ed3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32781 } } }
I20260926 11:34:21.971094 22883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82785531e6ec47f0a5270860e5cb7ed3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:34:21.971097 22932 mvcc.cc:205] Tried to move back new op lower bound from 7333570404196777984 to 7333570404133289984. Current Snapshot: MvccSnapshot[applied={T|T < 7333570404196777984}]
I20260926 11:34:21.981482 22919 catalog_manager.cc:1347] Generated new cluster ID: 891d4fb0da2443cd9aa02ddcb71f17dc
I20260926 11:34:21.981563 22919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:34:21.991106 22726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.958833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56188 (local address 127.0.98.62:33119)
0926 11:34:21.958960 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:21.958964 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:21.972115 (+ 13151us) server_negotiation.cc:407] Connection header received
0926 11:34:21.972312 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:21.972316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:21.972363 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:21.972412 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:21.973473 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.974243 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.975483 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.975902 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.976003 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:21.980410 (+ 4407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:21.980414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:21.980417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:21.980455 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:21.982221 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:21.987818 (+ 5597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:21.987822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:21.987824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:21.987918 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:21.989280 (+ 1362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:21.989283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:21.989285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:21.989488 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:21.989593 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:21.990811 (+ 1218us) server_negotiation.cc:299] Negotiation successful
0926 11:34:21.990931 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 11:34:21.996102 22934 mvcc.cc:205] Tried to move back new op lower bound from 7333570404196777984 to 7333570404133289984. Current Snapshot: MvccSnapshot[applied={T|T < 7333570404196777984}]
I20260926 11:34:21.999369 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.959579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57820 (local address 127.0.98.60:32781)
0926 11:34:21.959677 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:21.959681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:21.976871 (+ 17190us) server_negotiation.cc:407] Connection header received
0926 11:34:21.976988 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:21.976992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:21.977044 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:21.977115 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:21.978024 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.978743 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.980225 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.980564 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.980650 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:21.993602 (+ 12952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:21.993607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:21.993609 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:21.993643 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:21.995401 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:21.995931 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:21.995935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:21.995936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:21.996038 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:21.997309 (+ 1271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:21.997312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:21.997314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:21.997516 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:21.997619 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:21.999084 (+ 1465us) server_negotiation.cc:299] Negotiation successful
0926 11:34:21.999204 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 11:34:22.004348 22803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:21.960179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36456 (local address 127.0.98.61:42081)
0926 11:34:21.960276 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:21.960282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:21.979898 (+ 19616us) server_negotiation.cc:407] Connection header received
0926 11:34:21.980010 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:21.980014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:21.980069 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:21.980137 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:21.989872 (+ 9735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.990643 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.996349 (+ 5706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:21.996687 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:21.996767 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:22.001059 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:22.001063 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:22.001066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:22.001095 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:22.002663 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:22.003121 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:22.003125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:22.003127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:22.003203 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:22.003446 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:22.003449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:22.003451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:22.003645 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:22.003772 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:22.004084 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 11:34:22.004198 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 11:34:22.010025 22890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1557a1f8a10e4adead57ff1979cdc2e4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1557a1f8a10e4adead57ff1979cdc2e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1557a1f8a10e4adead57ff1979cdc2e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33119 } } peers { permanent_uuid: "45be645b1f704cf38cebf93d90884be6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42081 } } peers { permanent_uuid: "82785531e6ec47f0a5270860e5cb7ed3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32781 } } }
I20260926 11:34:22.010170 22890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1557a1f8a10e4adead57ff1979cdc2e4 [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:22.011298 22907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45be645b1f704cf38cebf93d90884be6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1557a1f8a10e4adead57ff1979cdc2e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1557a1f8a10e4adead57ff1979cdc2e4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33119 } } peers { permanent_uuid: "45be645b1f704cf38cebf93d90884be6" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42081 } } peers { permanent_uuid: "82785531e6ec47f0a5270860e5cb7ed3" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32781 } } }
I20260926 11:34:22.011422 22907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45be645b1f704cf38cebf93d90884be6 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:34:22.011777 22933 mvcc.cc:205] Tried to move back new op lower bound from 7333570404196777984 to 7333570404133289984. Current Snapshot: MvccSnapshot[applied={T|T < 7333570404196777984}]
I20260926 11:34:22.025657 22919 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:34:22.026989 22919 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:34:22.046732 22919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1557a1f8a10e4adead57ff1979cdc2e4: Generated new TSK 0
I20260926 11:34:22.047770 22919 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 11:34:22.080694 22803 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:34:22.077589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36470 (local address 127.0.98.61:42081)
0926 11:34:22.077720 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:22.077725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:22.079499 (+ 1774us) server_negotiation.cc:407] Connection header received
0926 11:34:22.079751 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:22.079755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:22.079809 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:22.079865 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:22.080546 (+ 681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36470: BlockingRecv error: recv got EOF from 127.0.0.1:36470 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 11:34:22.082273 22879 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:34:22.078136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57826 (local address 127.0.98.60:32781)
0926 11:34:22.078272 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:22.078276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:22.081192 (+ 2916us) server_negotiation.cc:407] Connection header received
0926 11:34:22.081361 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:22.081365 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:22.081414 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:22.081465 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:22.081971 (+ 506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57826: BlockingRecv error: recv got EOF from 127.0.0.1:57826 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 11:34:22.083653 22726 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:34:22.078318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56202 (local address 127.0.98.62:33119)
0926 11:34:22.078433 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:22.078437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:22.082779 (+ 4342us) server_negotiation.cc:407] Connection header received
0926 11:34:22.082944 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:22.082947 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:22.082997 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:22.083052 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:22.083520 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56202: BlockingRecv error: recv got EOF from 127.0.0.1:56202 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 11:34:22.088290 22803 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:34:22.086618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36484 (local address 127.0.98.61:42081)
0926 11:34:22.086729 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:22.086733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:22.087283 (+ 550us) server_negotiation.cc:407] Connection header received
0926 11:34:22.087507 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:22.087511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:22.087602 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:22.087654 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:22.088180 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36484: BlockingRecv error: recv got EOF from 127.0.0.1:36484 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 11:34:22.093369 22803 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:34:22.091910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36496 (local address 127.0.98.61:42081)
0926 11:34:22.092014 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:22.092018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:22.092484 (+ 466us) server_negotiation.cc:407] Connection header received
0926 11:34:22.092679 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:22.092683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:22.092735 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:22.092787 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:22.093268 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36496: BlockingRecv error: recv got EOF from 127.0.0.1:36496 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 11:34:22.098913 22803 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:34:22.097399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36500 (local address 127.0.98.61:42081)
0926 11:34:22.097496 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:22.097500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:22.098053 (+ 553us) server_negotiation.cc:407] Connection header received
0926 11:34:22.098231 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:22.098234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:22.098282 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:22.098331 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:22.098808 (+ 477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36500: BlockingRecv error: recv got EOF from 127.0.0.1:36500 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 11:34:22.106805 22803 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:34:22.105125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36506 (local address 127.0.98.61:42081)
0926 11:34:22.105266 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:22.105272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:22.105893 (+ 621us) server_negotiation.cc:407] Connection header received
0926 11:34:22.106075 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:22.106079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:22.106129 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:22.106179 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:22.106697 (+ 518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36506: BlockingRecv error: recv got EOF from 127.0.0.1:36506 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 11:34:22.119261 22803 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:34:22.117365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36522 (local address 127.0.98.61:42081)
0926 11:34:22.117490 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:22.117495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:22.118173 (+ 678us) server_negotiation.cc:407] Connection header received
0926 11:34:22.118389 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:22.118393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:22.118450 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:22.118507 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:22.119109 (+ 602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36522: BlockingRecv error: recv got EOF from 127.0.0.1:36522 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 11:34:22.140976 22803 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:34:22.139010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36526 (local address 127.0.98.61:42081)
0926 11:34:22.139158 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:22.139163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:22.139884 (+ 721us) server_negotiation.cc:407] Connection header received
0926 11:34:22.140147 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:22.140151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:22.140209 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:22.140270 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:22.140856 (+ 586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36526: BlockingRecv error: recv got EOF from 127.0.0.1:36526 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 11:34:22.176868 22803 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:34:22.175536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36532 (local address 127.0.98.61:42081)
0926 11:34:22.175662 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:22.175667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:22.175953 (+ 286us) server_negotiation.cc:407] Connection header received
0926 11:34:22.176133 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:22.176137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:22.176192 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:22.176283 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:22.176744 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36532: BlockingRecv error: recv got EOF from 127.0.0.1:36532 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 11:34:22.244367 22803 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:34:22.243069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36548 (local address 127.0.98.61:42081)
0926 11:34:22.243255 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:22.243259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:22.243387 (+ 128us) server_negotiation.cc:407] Connection header received
0926 11:34:22.243573 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:22.243579 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:22.243656 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:22.243760 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:22.244259 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36548: BlockingRecv error: recv got EOF from 127.0.0.1:36548 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260926 11:34:22.375844 22803 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:34:22.374597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36558 (local address 127.0.98.61:42081)
0926 11:34:22.374753 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:22.374758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:22.374883 (+ 125us) server_negotiation.cc:407] Connection header received
0926 11:34:22.375081 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:22.375084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:22.375132 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:22.375225 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:22.375688 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36558: BlockingRecv error: recv got EOF from 127.0.0.1:36558 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 11:34:22.635229 22803 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:34:22.634018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36574 (local address 127.0.98.61:42081)
0926 11:34:22.634160 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:22.634165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:22.634275 (+ 110us) server_negotiation.cc:407] Connection header received
0926 11:34:22.634460 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:22.634463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:22.634510 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:22.634616 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:22.635087 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36574: BlockingRecv error: recv got EOF from 127.0.0.1:36574 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 11:34:22.948498 22895 catalog_manager.cc:1259] Loaded cluster ID: 891d4fb0da2443cd9aa02ddcb71f17dc
I20260926 11:34:22.948587 22895 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 82785531e6ec47f0a5270860e5cb7ed3: loading cluster ID for follower catalog manager: success
I20260926 11:34:22.950294 22895 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 82785531e6ec47f0a5270860e5cb7ed3: acquiring CA information for follower catalog manager: success
I20260926 11:34:22.951397 22895 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 82785531e6ec47f0a5270860e5cb7ed3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 11:34:22.958329 22922 catalog_manager.cc:1259] Loaded cluster ID: 891d4fb0da2443cd9aa02ddcb71f17dc
I20260926 11:34:22.958405 22922 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 45be645b1f704cf38cebf93d90884be6: loading cluster ID for follower catalog manager: success
I20260926 11:34:22.960013 22922 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 45be645b1f704cf38cebf93d90884be6: acquiring CA information for follower catalog manager: success
I20260926 11:34:22.961122 22922 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 45be645b1f704cf38cebf93d90884be6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 11:34:23.153427 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:23.149582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36576 (local address 127.0.98.61:42081)
0926 11:34:23.149799 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:23.149802 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:23.149845 (+ 43us) server_negotiation.cc:407] Connection header received
0926 11:34:23.149987 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:23.149990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:23.150118 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:23.150237 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:34:23.150475 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:23.151007 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:23.151913 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:23.152158 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:23.152247 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:23.152534 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:34:23.152977 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:34:23.153083 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:23.153142 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 11:34:23.153196 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":71,"threads_started":1}
I20260926 11:34:23.160749 23011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:23.156967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57834 (local address 127.0.98.60:32781)
0926 11:34:23.157210 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:23.157213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:23.157228 (+ 15us) server_negotiation.cc:407] Connection header received
0926 11:34:23.157344 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:23.157347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:23.157463 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:23.157562 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:34:23.157832 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:23.158348 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:23.159157 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:23.159371 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:23.159452 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:23.159716 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:34:23.160210 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:34:23.160304 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:23.160477 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 11:34:23.160580 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":75,"threads_started":1}
I20260926 11:34:23.167845 23011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:23.164154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57844 (local address 127.0.98.60:32781)
0926 11:34:23.164251 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:23.164255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:23.164513 (+ 258us) server_negotiation.cc:407] Connection header received
0926 11:34:23.164683 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:23.164688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:23.164738 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:23.164804 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:34:23.165169 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:23.165904 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:23.166799 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:23.167059 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:23.167164 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:23.167365 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:34:23.167451 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:34:23.167518 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:23.167617 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 11:34:23.167665 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 11:34:23.168308 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:23.163872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36590 (local address 127.0.98.61:42081)
0926 11:34:23.164138 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:23.164144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:23.164380 (+ 236us) server_negotiation.cc:407] Connection header received
0926 11:34:23.164523 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:23.164526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:23.164575 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:23.164654 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:34:23.164943 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:23.165791 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:23.167050 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:23.167390 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:23.167498 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:23.167824 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:34:23.167914 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:34:23.167965 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:23.168130 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 11:34:23.168192 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260926 11:34:23.169486 23018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:23.163877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56206 (local address 127.0.98.62:33119)
0926 11:34:23.164279 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:23.164283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:23.164298 (+ 15us) server_negotiation.cc:407] Connection header received
0926 11:34:23.164509 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:23.164512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:23.164698 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:23.164811 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:34:23.165057 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:23.165638 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:23.167574 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:23.167951 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:23.168058 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:23.168466 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:34:23.168972 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:34:23.169045 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:23.169270 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 11:34:23.169345 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":88,"threads_started":1}
I20260926 11:34:23.172654 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 22654
I20260926 11:34:23.179278 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 22729
I20260926 11:34:23.184912 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 22804
2026-09-26T11:34:23Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3071 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 11:34:23 dist-test-slave-mrck krb5kdc[23022](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 11:34:23 dist-test-slave-mrck krb5kdc[23022](info): set up 2 sockets
Sep 26 11:34:23 dist-test-slave-mrck krb5kdc[23022](info): commencing operation
krb5kdc: starting...
W20260926 11:34:24.216176 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/krb5kdc/test-user.keytab.
Sep 26 11:34:24 dist-test-slave-mrck krb5kdc[23022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T11:34:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T11:34:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 11:34:24.338932 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42743
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42743,127.0.98.61:43369,127.0.98.60:35251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 11:34:24.449193 23035 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 11:34:24.449530 23035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:34:24.449605 23035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:34:24.449687 23035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:34:24.453634 23035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:34:24.453730 23035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:34:24.453768 23035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:34:24.453802 23035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:34:24.453835 23035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:34:24.459057 23035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42743,127.0.98.61:43369,127.0.98.60:35251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:34:24.460171 23035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:34:24.461176 23035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:34:24.469794 23041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:34:24.469827 23043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:34:24.469902 23035 server_base.cc:1034] running on GCE node
W20260926 11:34:24.469806 23040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:34:24.470573 23035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:34:24.471787 23035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:34:24.472990 23035 hybrid_clock.cc:648] HybridClock initialized: now 1790422464472951 us; error 54 us; skew 500 ppm
Sep 26 11:34:24 dist-test-slave-mrck krb5kdc[23022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:34:24.481469 23035 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 11:34:24.482435 23035 webserver.cc:460] Webserver started at http://127.0.98.62:34919/ using document root <none> and password file <none>
I20260926 11:34:24.483033 23035 fs_manager.cc:359] Metadata directory not provided
I20260926 11:34:24.483132 23035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:34:24.483366 23035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:34:24.485487 23035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-0/data/instance:
uuid: "eb52c8bd23a349cfa38c652bbcb1c044"
format_stamp: "Formatted at 2026-09-26 11:34:24 on dist-test-slave-mrck"
server_key: "a143548edd920aa89bb56bb471a2bf5d"
server_key_iv: "b7905e1e5fc6494865988465b84397da"
server_key_version: "encryptionkey@0"
I20260926 11:34:24.486066 23035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-0/wal/instance:
uuid: "eb52c8bd23a349cfa38c652bbcb1c044"
format_stamp: "Formatted at 2026-09-26 11:34:24 on dist-test-slave-mrck"
server_key: "a143548edd920aa89bb56bb471a2bf5d"
server_key_iv: "b7905e1e5fc6494865988465b84397da"
server_key_version: "encryptionkey@0"
I20260926 11:34:24.489930 23035 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260926 11:34:24.492372 23050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:34:24.493961 23035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 11:34:24.494068 23035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-0/wal
uuid: "eb52c8bd23a349cfa38c652bbcb1c044"
format_stamp: "Formatted at 2026-09-26 11:34:24 on dist-test-slave-mrck"
server_key: "a143548edd920aa89bb56bb471a2bf5d"
server_key_iv: "b7905e1e5fc6494865988465b84397da"
server_key_version: "encryptionkey@0"
I20260926 11:34:24.494158 23035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:34:24.518431 23035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:34:24.521835 23035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:34:24.521996 23035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:34:24.528810 23035 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42743
I20260926 11:34:24.529119 23102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42743 every 8 connection(s)
I20260926 11:34:24.529791 23035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-0/data/info.pb
I20260926 11:34:24.532490 23103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:34:24.535647 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 23035
I20260926 11:34:24.535807 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-0/wal/instance
I20260926 11:34:24.536087 392 external_mini_cluster.cc:1428] Setting key 8b697ea4f7b82082b19f419e5b889577
I20260926 11:34:24.536207 23103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 26 11:34:24 dist-test-slave-mrck krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:34:24.552443 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.538447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42743 (local address 127.0.0.1:50208)
0926 11:34:24.538910 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:24.538962 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 11:34:24.539977 (+ 1015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:34:24.540822 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:34:24.540826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:34:24.541149 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:34:24.541470 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:24.541483 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:24.543286 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:24.543290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:24.544140 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:24.544149 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:24.544762 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:24.544765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:24.544885 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:24.545496 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:34:24.545512 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:34:24.547332 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:34:24.549639 (+ 2307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:24.549648 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:24.549656 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:24.549868 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:24.550246 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:24.550250 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:24.550251 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:24.550378 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:24.551131 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:34:24.551138 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:34:24.551251 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 11:34:24.551382 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 11:34:24.551614 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":43,"thread_start_us":133,"threads_started":1}
I20260926 11:34:24.552448 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.539326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50208 (local address 127.0.98.62:42743)
0926 11:34:24.539663 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:24.539676 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:24.539748 (+ 72us) server_negotiation.cc:407] Connection header received
0926 11:34:24.540296 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:24.540304 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:24.540597 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:24.540816 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:24.541626 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:24.543062 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:24.544252 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:24.544631 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:24.544751 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:24.547474 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:24.547479 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:24.547489 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:24.547511 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:24.549514 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:24.549979 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:24.549988 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:24.549993 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:24.550135 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:24.550475 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:24.550481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:24.550485 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:24.551017 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:24.551173 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:24.551501 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 11:34:24.551688 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":200,"thread_start_us":88,"threads_started":1}
I20260926 11:34:24.555286 23103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43369 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:34:24.556123 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.555860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43369 (local address 127.0.0.1:35262)
0926 11:34:24.555980 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:24.556058 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 11:34:24.556357 23051 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43369: connect: Connection refused (error 111)
I20260926 11:34:24.557204 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.556996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43369 (local address 127.0.0.1:35274)
0926 11:34:24.557100 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:24.557154 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 11:34:24.557394 23103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43369: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43369: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 11:34:24.562083 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43369
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42743,127.0.98.61:43369,127.0.98.60:35251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 11:34:24.606662 23103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43369 } attempt: 1
I20260926 11:34:24.607517 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.607219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43369 (local address 127.0.0.1:35278)
0926 11:34:24.607367 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:24.607438 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 11:34:24.608330 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.608110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43369 (local address 127.0.0.1:35280)
0926 11:34:24.608225 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:24.608278 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 11:34:24.608482 23103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43369: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43369: connect: Connection refused (error 111)
W20260926 11:34:24.671703 23110 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 11:34:24.672076 23110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:34:24.672153 23110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:34:24.672231 23110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 11:34:24.676393 23110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:34:24.676457 23110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:34:24.676478 23110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:34:24.676496 23110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:34:24.676546 23110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:34:24.676669 23103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43369 } attempt: 2
I20260926 11:34:24.677513 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.677213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43369 (local address 127.0.0.1:35290)
0926 11:34:24.677364 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:24.677439 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 11:34:24.678251 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.678036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43369 (local address 127.0.0.1:35292)
0926 11:34:24.678140 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:24.678189 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 11:34:24.678434 23103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43369: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43369: connect: Connection refused (error 111)
I20260926 11:34:24.683005 23110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42743,127.0.98.61:43369,127.0.98.60:35251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:34:24.684257 23110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:34:24.685712 23110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 11:34:24.695096 23116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:34:24.695099 23118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:34:24.695099 23115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:34:24.695423 23110 server_base.cc:1034] running on GCE node
I20260926 11:34:24.696166 23110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:34:24.697358 23110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:34:24.698611 23110 hybrid_clock.cc:648] HybridClock initialized: now 1790422464698594 us; error 29 us; skew 500 ppm
Sep 26 11:34:24 dist-test-slave-mrck krb5kdc[23022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:34:24.701442 23110 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 11:34:24.702394 23110 webserver.cc:460] Webserver started at http://127.0.98.61:43955/ using document root <none> and password file <none>
I20260926 11:34:24.702972 23110 fs_manager.cc:359] Metadata directory not provided
I20260926 11:34:24.703066 23110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:34:24.703332 23110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:34:24.705466 23110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-1/data/instance:
uuid: "7a0e09799e7a43299d6e85f35e540def"
format_stamp: "Formatted at 2026-09-26 11:34:24 on dist-test-slave-mrck"
server_key: "b10660e8ac0034339e746cecb2afdbc2"
server_key_iv: "7c0787bebfd41a0efb710ddd086cc6c1"
server_key_version: "encryptionkey@0"
I20260926 11:34:24.706036 23110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-1/wal/instance:
uuid: "7a0e09799e7a43299d6e85f35e540def"
format_stamp: "Formatted at 2026-09-26 11:34:24 on dist-test-slave-mrck"
server_key: "b10660e8ac0034339e746cecb2afdbc2"
server_key_iv: "7c0787bebfd41a0efb710ddd086cc6c1"
server_key_version: "encryptionkey@0"
I20260926 11:34:24.709820 23110 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 11:34:24.712289 23125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:34:24.713596 23110 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 11:34:24.713774 23110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-1/wal
uuid: "7a0e09799e7a43299d6e85f35e540def"
format_stamp: "Formatted at 2026-09-26 11:34:24 on dist-test-slave-mrck"
server_key: "b10660e8ac0034339e746cecb2afdbc2"
server_key_iv: "7c0787bebfd41a0efb710ddd086cc6c1"
server_key_version: "encryptionkey@0"
I20260926 11:34:24.713907 23110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:34:24.733753 23110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:34:24.737138 23110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:34:24.737352 23110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:34:24.744388 23110 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43369
I20260926 11:34:24.744693 23177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43369 every 8 connection(s)
I20260926 11:34:24.745406 23110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-1/data/info.pb
I20260926 11:34:24.748275 23178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:34:24.748562 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 23110
I20260926 11:34:24.748709 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-1/wal/instance
I20260926 11:34:24.748996 392 external_mini_cluster.cc:1428] Setting key 9b2c4ac2862a1e19b45e46c69885f1e8
I20260926 11:34:24.752841 23178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 26 11:34:24 dist-test-slave-mrck krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:34:24.766880 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.754858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50212 (local address 127.0.98.62:42743)
0926 11:34:24.754999 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:24.755003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:24.756125 (+ 1122us) server_negotiation.cc:407] Connection header received
0926 11:34:24.757113 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:24.757117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:24.757177 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:24.757280 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:24.758465 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:24.759154 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:24.760394 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:24.760743 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:24.760875 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:24.763261 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:24.763264 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:24.763266 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:24.763291 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:24.764743 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:24.765280 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:24.765283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:24.765285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:24.765372 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:24.765704 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:24.765709 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:24.765711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:24.765900 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:24.766011 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:24.766546 (+ 535us) server_negotiation.cc:299] Negotiation successful
0926 11:34:24.766703 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 11:34:24.767845 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.755365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42743 (local address 127.0.0.1:50212)
0926 11:34:24.755955 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:24.755999 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 11:34:24.756870 (+ 871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:34:24.757447 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:34:24.757454 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:34:24.757843 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:34:24.758317 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:24.758330 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:24.759283 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:24.759291 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:24.760261 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:24.760270 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:24.760911 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:24.760914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:24.761065 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:24.761669 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:34:24.761686 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:34:24.763074 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:34:24.764882 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:24.764888 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:24.764905 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:24.765173 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:24.765465 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:24.765470 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:24.765472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:24.765610 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:24.766008 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:34:24.766016 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:34:24.766287 (+ 271us) client_negotiation.cc:769] Sending connection context
0926 11:34:24.766478 (+ 191us) client_negotiation.cc:240] Negotiation successful
0926 11:34:24.766840 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":135,"threads_started":1}
I20260926 11:34:24.769423 23178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43369 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:34:24.769614 23103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43369 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 11:34:24.774521 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskF2HClO/build/debug/bin/kudu
/tmp/dist-test-taskF2HClO/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35251
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42743,127.0.98.61:43369,127.0.98.60:35251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 11:34:24 dist-test-slave-mrck krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 11:34:24 dist-test-slave-mrck krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 11:34:24.781075 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.770025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43369 (local address 127.0.0.1:35308)
0926 11:34:24.770485 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:24.770496 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 11:34:24.770628 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:34:24.771012 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:34:24.771015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:34:24.771227 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:34:24.771343 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:24.771348 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:24.772524 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:24.772530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:24.773092 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:24.773099 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:24.773866 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:24.773872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:24.773992 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:24.774521 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:34:24.774537 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:34:24.776502 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:34:24.779428 (+ 2926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:24.779434 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:24.779437 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:24.779755 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:24.780041 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:24.780043 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:24.780045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:24.780115 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:24.780555 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:34:24.780558 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:34:24.780634 (+ 76us) client_negotiation.cc:769] Sending connection context
0926 11:34:24.780752 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 11:34:24.780883 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":2}
I20260926 11:34:24.781102 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.770158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35308 (local address 127.0.98.61:43369)
0926 11:34:24.770476 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:24.770490 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:24.770553 (+ 63us) server_negotiation.cc:407] Connection header received
0926 11:34:24.770733 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:24.770737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:24.770895 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:24.771039 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:24.771449 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:24.772422 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:24.773222 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:24.773742 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:24.774227 (+ 485us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:24.776649 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:24.776652 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:24.776654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:24.776675 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:24.779330 (+ 2655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:24.779864 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:24.779867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:24.779869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:24.779951 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:24.780200 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:24.780206 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:24.780208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:24.780450 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:24.780592 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:24.780832 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 11:34:24.780959 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":214,"spinlock_wait_cycles":3328,"thread_start_us":103,"threads_started":1}
I20260926 11:34:24.781459 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.770061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43369 (local address 127.0.0.1:35324)
0926 11:34:24.770225 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:24.770239 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 11:34:24.770349 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:34:24.771085 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:34:24.771088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:34:24.771277 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:34:24.771374 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:24.771379 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:24.772653 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:24.772657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:24.773331 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:24.773341 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:24.773910 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:24.773917 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:24.774027 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:24.774895 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:34:24.774914 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:34:24.776430 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:34:24.779351 (+ 2921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:24.779357 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:24.779362 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:24.779588 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:24.780019 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:24.780023 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:24.780025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:24.780134 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:24.780676 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:34:24.780682 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:34:24.780804 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 11:34:24.780902 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 11:34:24.781064 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 11:34:24.782510 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.770227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35324 (local address 127.0.98.61:43369)
0926 11:34:24.770888 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:24.770894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:24.770905 (+ 11us) server_negotiation.cc:407] Connection header received
0926 11:34:24.770930 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:24.770932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:24.770963 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:24.771065 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:24.771485 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:24.772462 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:24.773464 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:24.773794 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:24.775822 (+ 2028us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:24.776572 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:24.776585 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:24.776603 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:24.776633 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:24.779230 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:24.779704 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:24.779714 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:24.779715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:24.779903 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:24.780261 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:24.780267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:24.780271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:24.780544 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:24.782037 (+ 1493us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:24.782123 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 11:34:24.782252 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2109,"server-negotiator.queue_time_us":599,"spinlock_wait_cycles":3968,"thread_start_us":138,"threads_started":1}
I20260926 11:34:24.783210 23178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35251 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:34:24.783248 23103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35251 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:34:24.783980 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.783716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35251 (local address 127.0.0.1:36850)
0926 11:34:24.783872 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:24.783922 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 11:34:24.784960 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.784680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35251 (local address 127.0.0.1:36856)
0926 11:34:24.784804 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:24.784889 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 11:34:24.785053 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.784763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35251 (local address 127.0.0.1:36870)
0926 11:34:24.784920 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:24.784982 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260926 11:34:24.785214 23103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35251: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35251: connect: Connection refused (error 111)
W20260926 11:34:24.785310 23126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35251: connect: Connection refused (error 111)
I20260926 11:34:24.786262 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.786053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35251 (local address 127.0.0.1:36874)
0926 11:34:24.786169 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:24.786212 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 11:34:24.786490 23178 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35251: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35251: connect: Connection refused (error 111)
I20260926 11:34:24.816424 23103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35251 } attempt: 1
I20260926 11:34:24.817325 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.817020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35251 (local address 127.0.0.1:36888)
0926 11:34:24.817181 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:24.817245 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 11:34:24.818064 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.817838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35251 (local address 127.0.0.1:36904)
0926 11:34:24.817943 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:24.817993 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 11:34:24.818190 23103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35251: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35251: connect: Connection refused (error 111)
I20260926 11:34:24.835768 23178 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35251 } attempt: 1
I20260926 11:34:24.836740 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.836405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35251 (local address 127.0.0.1:36918)
0926 11:34:24.836589 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:24.836658 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260926 11:34:24.837517 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.837288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35251 (local address 127.0.0.1:36920)
0926 11:34:24.837401 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:24.837455 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 11:34:24.837652 23178 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35251: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35251: connect: Connection refused (error 111)
W20260926 11:34:24.890815 23185 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 11:34:24.891188 23185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 11:34:24.891263 23185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 11:34:24.891343 23185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 11:34:24.893388 23103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35251 } attempt: 2
I20260926 11:34:24.894244 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.893940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35251 (local address 127.0.0.1:36932)
0926 11:34:24.894091 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:24.894165 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 11:34:24.894946 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.894749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35251 (local address 127.0.0.1:36944)
0926 11:34:24.894846 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:24.894897 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 11:34:24.895074 23103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35251: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35251: connect: Connection refused (error 111)
W20260926 11:34:24.895854 23185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 11:34:24.895932 23185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 11:34:24.895979 23185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 11:34:24.896018 23185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 11:34:24.896057 23185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 11:34:24.901675 23185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42743,127.0.98.61:43369,127.0.98.60:35251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 10:47:36 UTC on ccycloud.quasar-emzlrj.root.comops.site
build id 4673837
I20260926 11:34:24.902812 23185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 11:34:24.903891 23185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 11:34:24.905830 23178 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35251 } attempt: 2
I20260926 11:34:24.906701 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.906414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35251 (local address 127.0.0.1:36954)
0926 11:34:24.906554 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:24.906624 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 11:34:24.907485 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.907257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35251 (local address 127.0.0.1:36970)
0926 11:34:24.907374 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:24.907423 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 11:34:24.907618 23178 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35251: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35251: connect: Connection refused (error 111)
W20260926 11:34:24.912542 23191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:34:24.912557 23192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 11:34:24.912549 23194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 11:34:24.912808 23185 server_base.cc:1034] running on GCE node
I20260926 11:34:24.913344 23185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 11:34:24.914448 23185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 11:34:24.915604 23185 hybrid_clock.cc:648] HybridClock initialized: now 1790422464915571 us; error 46 us; skew 500 ppm
Sep 26 11:34:24 dist-test-slave-mrck krb5kdc[23022](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 11:34:24.918329 23185 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 11:34:24.919260 23185 webserver.cc:460] Webserver started at http://127.0.98.60:37207/ using document root <none> and password file <none>
I20260926 11:34:24.919905 23185 fs_manager.cc:359] Metadata directory not provided
I20260926 11:34:24.919996 23185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 11:34:24.920228 23185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 11:34:24.922324 23185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-2/data/instance:
uuid: "7f91a563b093440fb3634fc7a721bc59"
format_stamp: "Formatted at 2026-09-26 11:34:24 on dist-test-slave-mrck"
server_key: "c254738a476f013d7aaba2d6c0764a2a"
server_key_iv: "c08165a0d817fc267fddb6cf75223f71"
server_key_version: "encryptionkey@0"
I20260926 11:34:24.922878 23185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-2/wal/instance:
uuid: "7f91a563b093440fb3634fc7a721bc59"
format_stamp: "Formatted at 2026-09-26 11:34:24 on dist-test-slave-mrck"
server_key: "c254738a476f013d7aaba2d6c0764a2a"
server_key_iv: "c08165a0d817fc267fddb6cf75223f71"
server_key_version: "encryptionkey@0"
I20260926 11:34:24.926703 23185 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 11:34:24.929405 23201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 11:34:24.930670 23185 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 11:34:24.930837 23185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-2/wal
uuid: "7f91a563b093440fb3634fc7a721bc59"
format_stamp: "Formatted at 2026-09-26 11:34:24 on dist-test-slave-mrck"
server_key: "c254738a476f013d7aaba2d6c0764a2a"
server_key_iv: "c08165a0d817fc267fddb6cf75223f71"
server_key_version: "encryptionkey@0"
I20260926 11:34:24.930964 23185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 11:34:24.958614 23185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 11:34:24.961912 23185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 11:34:24.962119 23185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 11:34:24.969631 23185 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35251
I20260926 11:34:24.969945 23253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35251 every 8 connection(s)
I20260926 11:34:24.970671 23185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-2/data/info.pb
I20260926 11:34:24.971926 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskF2HClO/build/debug/bin/kudu as pid 23185
I20260926 11:34:24.972052 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskF2HClO/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790422180210734-392-0/minicluster-data/master-2/wal/instance
I20260926 11:34:24.972347 392 external_mini_cluster.cc:1428] Setting key e87e59a06d452b17508188fcea5c6000
I20260926 11:34:24.973881 23254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 11:34:24.978597 23254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 11:34:24 dist-test-slave-mrck krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 11:34:24 dist-test-slave-mrck krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 11:34:24.987874 23257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.973785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36980 (local address 127.0.98.60:35251)
0926 11:34:24.974207 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:24.974230 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:24.974273 (+ 43us) server_negotiation.cc:407] Connection header received
0926 11:34:24.975130 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:24.975153 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:24.975592 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:24.975999 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:24.976669 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:24.978155 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:24.979033 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:24.979682 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:24.979856 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:24.982744 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:24.982750 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:24.982769 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:24.982794 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:24.984711 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:24.985218 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:24.985227 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:24.985233 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:24.985370 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:24.985763 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:24.985768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:24.985770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:24.986207 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:24.986417 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:24.986721 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 11:34:24.987012 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":139,"threads_started":1}
I20260926 11:34:24.990258 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.980598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50218 (local address 127.0.98.62:42743)
0926 11:34:24.980741 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:24.980744 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:24.981097 (+ 353us) server_negotiation.cc:407] Connection header received
0926 11:34:24.981267 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:24.981271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:24.981320 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:24.981416 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:24.982280 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:24.982806 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:24.984117 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:24.984335 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:24.984418 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:24.986599 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:24.986611 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:24.986614 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:24.986641 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:24.988364 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:24.988956 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:24.988961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:24.988964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:24.989056 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:24.989343 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:24.989345 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:24.989346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:24.989518 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:24.989590 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:24.989974 (+ 384us) server_negotiation.cc:299] Negotiation successful
0926 11:34:24.990096 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 11:34:24.990268 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.980550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42743 (local address 127.0.0.1:50218)
0926 11:34:24.980964 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:24.981004 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 11:34:24.981160 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:34:24.981464 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:34:24.981470 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:34:24.981948 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:34:24.982131 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:24.982139 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:24.982924 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:24.982931 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:24.983979 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:24.983990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:24.984452 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:24.984457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:24.984583 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:24.985058 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:34:24.985069 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:34:24.986461 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:34:24.988480 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:24.988487 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:24.988491 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:24.988856 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:24.989162 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:24.989165 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:24.989166 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:24.989258 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:24.989607 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:34:24.989620 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:34:24.989747 (+ 127us) client_negotiation.cc:769] Sending connection context
0926 11:34:24.989890 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 11:34:24.990089 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":101,"threads_started":1}
I20260926 11:34:24.991386 23254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43369 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:34:24.994254 23103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35251 } attempt: 3
Sep 26 11:34:24 dist-test-slave-mrck krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 11:34:24 dist-test-slave-mrck krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 11:34:24.998772 23178 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35251 } attempt: 3
Sep 26 11:34:24 dist-test-slave-mrck krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 11:34:25.002149 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.991988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35336 (local address 127.0.98.61:43369)
0926 11:34:24.992117 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:24.992122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:24.992137 (+ 15us) server_negotiation.cc:407] Connection header received
0926 11:34:24.992210 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:24.992214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:24.992271 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:24.992331 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:24.992735 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:24.993479 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:24.994455 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:24.994737 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:24.994817 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:24.997556 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:24.997560 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:24.997563 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:24.997597 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:25.000050 (+ 2453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:25.000841 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:25.000848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:25.000853 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:25.000941 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:25.001266 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:25.001270 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:25.001271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:25.001470 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:25.001537 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:25.001903 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 11:34:25.002010 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 11:34:25.002012 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.990711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50224 (local address 127.0.98.62:42743)
0926 11:34:24.990833 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:24.990839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:24.990850 (+ 11us) server_negotiation.cc:407] Connection header received
0926 11:34:24.990881 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:24.990884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:24.990921 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:24.990994 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:24.991698 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:24.992427 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:24.993509 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:24.993851 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:24.993956 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:24.997571 (+ 3615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:24.997575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:24.997579 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:24.997613 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:24.999435 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:24.999983 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:24.999987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:24.999989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:25.000072 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:25.000570 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:25.000576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:25.000578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:25.000817 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:25.001048 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:25.001725 (+ 677us) server_negotiation.cc:299] Negotiation successful
0926 11:34:25.001848 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 11:34:25.002599 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.991868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43369 (local address 127.0.0.1:35336)
0926 11:34:24.992040 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:24.992052 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 11:34:24.992129 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:34:24.992347 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:34:24.992350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:34:24.992520 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:34:24.992623 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:24.992628 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:24.993591 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:24.993594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:24.994311 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:24.994321 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:24.994959 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:24.994962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:24.995065 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:24.995565 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:34:24.995576 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:34:24.997341 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:34:25.000399 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:25.000405 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:25.000410 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:25.000718 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:25.001059 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:25.001065 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:25.001069 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:25.001156 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:25.001630 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:34:25.001633 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:34:25.001725 (+ 92us) client_negotiation.cc:769] Sending connection context
0926 11:34:25.002308 (+ 583us) client_negotiation.cc:240] Negotiation successful
0926 11:34:25.002435 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260926 11:34:25.003371 23254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35251 } has no permanent_uuid. Determining permanent_uuid...
I20260926 11:34:25.003763 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.994666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35251 (local address 127.0.0.1:36984)
0926 11:34:24.994760 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:24.994776 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:34:24.994850 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:34:24.995082 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:34:24.995087 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:34:24.995274 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:34:24.995389 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:24.995395 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:24.996340 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:24.996345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:24.996936 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:24.996944 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:24.997402 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:24.997405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:24.997486 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:24.997950 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:34:24.997960 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:34:24.999547 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:34:25.001987 (+ 2440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:25.001990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:25.001992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:25.002252 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:25.002735 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:25.002738 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:25.002739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:25.002822 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:25.003261 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:34:25.003265 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:34:25.003349 (+ 84us) client_negotiation.cc:769] Sending connection context
0926 11:34:25.003428 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 11:34:25.003573 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 11:34:25.004248 23257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.994748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36984 (local address 127.0.98.60:35251)
0926 11:34:24.994864 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:24.994868 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:24.994881 (+ 13us) server_negotiation.cc:407] Connection header received
0926 11:34:24.994918 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:24.994920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:24.994960 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:24.995037 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:24.995538 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:24.996228 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:24.997050 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:24.997290 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:24.997397 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:25.000046 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:25.000050 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:25.000052 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:25.000075 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:25.001863 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:25.002532 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:25.002535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:25.002537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:25.002628 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:25.002934 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:25.002940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:25.002942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:25.003163 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:25.003937 (+ 774us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:25.004008 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 11:34:25.004112 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 11:34:25 dist-test-slave-mrck krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 11:34:25.008500 23103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb52c8bd23a349cfa38c652bbcb1c044: Bootstrap starting.
Sep 26 11:34:25 dist-test-slave-mrck krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 11:34:25 dist-test-slave-mrck krb5kdc[23022](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790422464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 11:34:25.011385 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.999198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35251 (local address 127.0.0.1:36986)
0926 11:34:24.999456 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:24.999472 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 11:34:24.999542 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:34:25.000648 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:34:25.000652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:34:25.000821 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:34:25.000951 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:25.000959 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:25.003559 (+ 2600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:25.003563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:25.004401 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:25.004418 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:25.005376 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:25.005379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:25.005461 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:25.005941 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:34:25.005955 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:34:25.007477 (+ 1522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:34:25.009820 (+ 2343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:25.009824 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:25.009826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:25.010088 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:25.010421 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:25.010424 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:25.010426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:25.010509 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:25.010916 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:34:25.010919 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:34:25.011009 (+ 90us) client_negotiation.cc:769] Sending connection context
0926 11:34:25.011087 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 11:34:25.011231 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183}
I20260926 11:34:25.011713 23103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb52c8bd23a349cfa38c652bbcb1c044: Neither blocks nor log segments found. Creating new log.
I20260926 11:34:25.012382 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:24.999780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36986 (local address 127.0.98.60:35251)
0926 11:34:25.000369 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:25.000376 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:25.000387 (+ 11us) server_negotiation.cc:407] Connection header received
0926 11:34:25.000414 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:25.000416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:25.000481 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:25.001073 (+ 592us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:25.001197 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:25.003387 (+ 2190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:25.004848 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:25.005252 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:25.005679 (+ 427us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:25.007633 (+ 1954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:25.007637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:25.007639 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:25.007662 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:25.009685 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:25.010235 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:25.010238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:25.010240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:25.010320 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:25.010611 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:25.010613 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:25.010615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:25.010804 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:25.012019 (+ 1215us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:25.012095 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 11:34:25.012221 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":520,"thread_start_us":133,"threads_started":1}
I20260926 11:34:25.012709 23103 log.cc:824] T 00000000000000000000000000000000 P eb52c8bd23a349cfa38c652bbcb1c044: Log is configured to *not* fsync() on all Append() calls
I20260926 11:34:25.013540 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:25.004012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37002 (local address 127.0.98.60:35251)
0926 11:34:25.004298 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:25.004362 (+ 64us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:25.004374 (+ 12us) server_negotiation.cc:407] Connection header received
0926 11:34:25.004469 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:25.004473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:25.004511 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:25.004587 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:25.005074 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:25.005913 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:25.006873 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:25.007210 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:25.007294 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:25.009660 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:25.009664 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:25.009668 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:25.009693 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:25.011467 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:25.012223 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:25.012226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:25.012227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:25.012310 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:25.012588 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:25.012591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:25.012593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:25.012792 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:25.012920 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:25.013251 (+ 331us) server_negotiation.cc:299] Negotiation successful
0926 11:34:25.013349 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":229,"thread_start_us":124,"threads_started":1}
I20260926 11:34:25.013948 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:25.003885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35251 (local address 127.0.0.1:37002)
0926 11:34:25.004319 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0926 11:34:25.004334 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 11:34:25.004398 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 11:34:25.004724 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 11:34:25.004729 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 11:34:25.004894 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 11:34:25.004967 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:25.004973 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:25.006011 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:25.006014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:25.006771 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 11:34:25.006779 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 11:34:25.007379 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 11:34:25.007382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 11:34:25.007449 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:25.007940 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 11:34:25.007955 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 11:34:25.009510 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 11:34:25.011562 (+ 2052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:25.011567 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:25.011572 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:25.012113 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:25.012411 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 11:34:25.012414 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 11:34:25.012417 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 11:34:25.012501 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 11:34:25.012904 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 11:34:25.012909 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 11:34:25.013002 (+ 93us) client_negotiation.cc:769] Sending connection context
0926 11:34:25.013106 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 11:34:25.013780 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"spinlock_wait_cycles":2944}
I20260926 11:34:25.015316 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:25.003934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35352 (local address 127.0.98.61:43369)
0926 11:34:25.004032 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:25.004036 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:25.004244 (+ 208us) server_negotiation.cc:407] Connection header received
0926 11:34:25.004650 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:25.004655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:25.004698 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:25.004746 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:25.005779 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:25.006510 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:25.007652 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:25.008106 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:25.008187 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:25.011495 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:25.011501 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:25.011508 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:25.011535 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:25.013522 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:25.014206 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:25.014212 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:25.014217 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:25.014299 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:25.014558 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:25.014564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:25.014566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:25.014787 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:25.014946 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:25.015072 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 11:34:25.015180 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 11:34:25.015612 23178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a0e09799e7a43299d6e85f35e540def: Bootstrap starting.
I20260926 11:34:25.018131 23178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a0e09799e7a43299d6e85f35e540def: Neither blocks nor log segments found. Creating new log.
I20260926 11:34:25.018790 23178 log.cc:824] T 00000000000000000000000000000000 P 7a0e09799e7a43299d6e85f35e540def: Log is configured to *not* fsync() on all Append() calls
I20260926 11:34:25.020932 23178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a0e09799e7a43299d6e85f35e540def: No bootstrap required, opened a new log
I20260926 11:34:25.022063 23254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f91a563b093440fb3634fc7a721bc59: Bootstrap starting.
I20260926 11:34:25.024060 23178 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a0e09799e7a43299d6e85f35e540def [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb52c8bd23a349cfa38c652bbcb1c044" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } } peers { permanent_uuid: "7a0e09799e7a43299d6e85f35e540def" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43369 } } peers { permanent_uuid: "7f91a563b093440fb3634fc7a721bc59" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35251 } }
I20260926 11:34:25.024305 23178 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a0e09799e7a43299d6e85f35e540def [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a0e09799e7a43299d6e85f35e540def, State: Initialized, Role: FOLLOWER
I20260926 11:34:25.024497 23254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f91a563b093440fb3634fc7a721bc59: Neither blocks nor log segments found. Creating new log.
I20260926 11:34:25.024627 23178 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a0e09799e7a43299d6e85f35e540def [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb52c8bd23a349cfa38c652bbcb1c044" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } } peers { permanent_uuid: "7a0e09799e7a43299d6e85f35e540def" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43369 } } peers { permanent_uuid: "7f91a563b093440fb3634fc7a721bc59" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35251 } }
I20260926 11:34:25.025374 23254 log.cc:824] T 00000000000000000000000000000000 P 7f91a563b093440fb3634fc7a721bc59: Log is configured to *not* fsync() on all Append() calls
I20260926 11:34:25.025720 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 11:34:25.025734 23103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb52c8bd23a349cfa38c652bbcb1c044: No bootstrap required, opened a new log
I20260926 11:34:25.025983 23178 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a0e09799e7a43299d6e85f35e540def [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:34:25.026969 23077 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "eb52c8bd23a349cfa38c652bbcb1c044"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50224
I20260926 11:34:25.028343 23254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f91a563b093440fb3634fc7a721bc59: No bootstrap required, opened a new log
I20260926 11:34:25.030220 23103 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb52c8bd23a349cfa38c652bbcb1c044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb52c8bd23a349cfa38c652bbcb1c044" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } } peers { permanent_uuid: "7a0e09799e7a43299d6e85f35e540def" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43369 } } peers { permanent_uuid: "7f91a563b093440fb3634fc7a721bc59" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35251 } }
I20260926 11:34:25.030468 23103 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb52c8bd23a349cfa38c652bbcb1c044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb52c8bd23a349cfa38c652bbcb1c044, State: Initialized, Role: FOLLOWER
I20260926 11:34:25.030818 23103 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb52c8bd23a349cfa38c652bbcb1c044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb52c8bd23a349cfa38c652bbcb1c044" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } } peers { permanent_uuid: "7a0e09799e7a43299d6e85f35e540def" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43369 } } peers { permanent_uuid: "7f91a563b093440fb3634fc7a721bc59" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35251 } }
I20260926 11:34:25.031369 23276 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb52c8bd23a349cfa38c652bbcb1c044 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb52c8bd23a349cfa38c652bbcb1c044" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } } peers { permanent_uuid: "7a0e09799e7a43299d6e85f35e540def" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43369 } } peers { permanent_uuid: "7f91a563b093440fb3634fc7a721bc59" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35251 } } }
I20260926 11:34:25.031682 23276 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb52c8bd23a349cfa38c652bbcb1c044 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:34:25.032246 23103 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb52c8bd23a349cfa38c652bbcb1c044 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:34:25.032292 23077 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb52c8bd23a349cfa38c652bbcb1c044 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 11:34:25.032394 23077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb52c8bd23a349cfa38c652bbcb1c044 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:34:25.032811 23254 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f91a563b093440fb3634fc7a721bc59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb52c8bd23a349cfa38c652bbcb1c044" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } } peers { permanent_uuid: "7a0e09799e7a43299d6e85f35e540def" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43369 } } peers { permanent_uuid: "7f91a563b093440fb3634fc7a721bc59" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35251 } }
I20260926 11:34:25.033017 23254 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f91a563b093440fb3634fc7a721bc59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f91a563b093440fb3634fc7a721bc59, State: Initialized, Role: FOLLOWER
I20260926 11:34:25.030061 23110 master_runner.cc:186] Error getting master registration for 127.0.98.62:42743: OK, instance_id { permanent_uuid: "eb52c8bd23a349cfa38c652bbcb1c044" instance_seqno: 1790422464526903 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 11:34:25.033366 23110 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 11:34:25.033309 23254 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f91a563b093440fb3634fc7a721bc59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb52c8bd23a349cfa38c652bbcb1c044" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } } peers { permanent_uuid: "7a0e09799e7a43299d6e85f35e540def" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43369 } } peers { permanent_uuid: "7f91a563b093440fb3634fc7a721bc59" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35251 } }
I20260926 11:34:25.033818 23077 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb52c8bd23a349cfa38c652bbcb1c044 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb52c8bd23a349cfa38c652bbcb1c044" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } } peers { permanent_uuid: "7a0e09799e7a43299d6e85f35e540def" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43369 } } peers { permanent_uuid: "7f91a563b093440fb3634fc7a721bc59" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35251 } }
I20260926 11:34:25.034565 23254 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f91a563b093440fb3634fc7a721bc59 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 11:34:25.034847 23077 leader_election.cc:288] T 00000000000000000000000000000000 P eb52c8bd23a349cfa38c652bbcb1c044 [CANDIDATE]: Term 1 election: Requested vote from peers 7a0e09799e7a43299d6e85f35e540def (127.0.98.61:43369), 7f91a563b093440fb3634fc7a721bc59 (127.0.98.60:35251)
I20260926 11:34:25.035972 23228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb52c8bd23a349cfa38c652bbcb1c044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7f91a563b093440fb3634fc7a721bc59"
I20260926 11:34:25.036253 23228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f91a563b093440fb3634fc7a721bc59 [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:34:25.037663 23228 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7f91a563b093440fb3634fc7a721bc59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb52c8bd23a349cfa38c652bbcb1c044 in term 1.
I20260926 11:34:25.037654 23152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "eb52c8bd23a349cfa38c652bbcb1c044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7a0e09799e7a43299d6e85f35e540def"
I20260926 11:34:25.037891 23152 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a0e09799e7a43299d6e85f35e540def [term 0 FOLLOWER]: Advancing to term 1
I20260926 11:34:25.038151 23051 leader_election.cc:302] T 00000000000000000000000000000000 P eb52c8bd23a349cfa38c652bbcb1c044 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f91a563b093440fb3634fc7a721bc59, eb52c8bd23a349cfa38c652bbcb1c044; no voters:
I20260926 11:34:25.038893 23282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f91a563b093440fb3634fc7a721bc59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb52c8bd23a349cfa38c652bbcb1c044" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } } peers { permanent_uuid: "7a0e09799e7a43299d6e85f35e540def" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43369 } } peers { permanent_uuid: "7f91a563b093440fb3634fc7a721bc59" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35251 } } }
I20260926 11:34:25.039057 23282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f91a563b093440fb3634fc7a721bc59 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:34:25.039136 23152 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a0e09799e7a43299d6e85f35e540def [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb52c8bd23a349cfa38c652bbcb1c044 in term 1.
W20260926 11:34:25.039222 23275 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a0e09799e7a43299d6e85f35e540def: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:34:25.039299 23275 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:34:25.039666 23276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb52c8bd23a349cfa38c652bbcb1c044 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 11:34:25.039923 23276 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb52c8bd23a349cfa38c652bbcb1c044 [term 1 LEADER]: Becoming Leader. State: Replica: eb52c8bd23a349cfa38c652bbcb1c044, State: Running, Role: LEADER
I20260926 11:34:25.040282 23276 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb52c8bd23a349cfa38c652bbcb1c044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb52c8bd23a349cfa38c652bbcb1c044" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } } peers { permanent_uuid: "7a0e09799e7a43299d6e85f35e540def" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43369 } } peers { permanent_uuid: "7f91a563b093440fb3634fc7a721bc59" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35251 } }
I20260926 11:34:25.042071 23276 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb52c8bd23a349cfa38c652bbcb1c044 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb52c8bd23a349cfa38c652bbcb1c044. Latest consensus state: current_term: 1 leader_uuid: "eb52c8bd23a349cfa38c652bbcb1c044" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb52c8bd23a349cfa38c652bbcb1c044" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } } peers { permanent_uuid: "7a0e09799e7a43299d6e85f35e540def" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43369 } } peers { permanent_uuid: "7f91a563b093440fb3634fc7a721bc59" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35251 } } }
I20260926 11:34:25.042196 23276 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb52c8bd23a349cfa38c652bbcb1c044 [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:25.044656 23035 master_runner.cc:263] No leader master found from master eb52c8bd23a349cfa38c652bbcb1c044
I20260926 11:34:25.044723 23035 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 11:34:25.044921 23263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a0e09799e7a43299d6e85f35e540def [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb52c8bd23a349cfa38c652bbcb1c044" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } } peers { permanent_uuid: "7a0e09799e7a43299d6e85f35e540def" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43369 } } peers { permanent_uuid: "7f91a563b093440fb3634fc7a721bc59" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35251 } } }
I20260926 11:34:25.045074 23263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a0e09799e7a43299d6e85f35e540def [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:34:25.045397 23310 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 11:34:25.049280 23310 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260926 11:34:25.052989 23304 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7f91a563b093440fb3634fc7a721bc59: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 11:34:25.053086 23304 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 11:34:25.059150 23228 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7f91a563b093440fb3634fc7a721bc59 [term 1 FOLLOWER]: Refusing update from remote peer eb52c8bd23a349cfa38c652bbcb1c044: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:34:25.059938 23260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:25.042732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37010 (local address 127.0.98.60:35251)
0926 11:34:25.042844 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:25.042849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:25.050153 (+ 7304us) server_negotiation.cc:407] Connection header received
0926 11:34:25.050262 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:25.050265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:25.050319 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:25.050407 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:25.051463 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:25.052252 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:25.053565 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:25.053921 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:25.054015 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:25.055886 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:25.055890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:25.055893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:25.055922 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:25.057769 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:25.058290 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:25.058294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:25.058296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:25.058382 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:25.058697 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:25.058700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:25.058702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:25.058945 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:25.059090 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:25.059583 (+ 493us) server_negotiation.cc:299] Negotiation successful
0926 11:34:25.059718 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 11:34:25.059931 23276 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eb52c8bd23a349cfa38c652bbcb1c044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f91a563b093440fb3634fc7a721bc59" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:34:25.062178 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:25.039867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50234 (local address 127.0.98.62:42743)
0926 11:34:25.039999 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:25.040006 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:25.043213 (+ 3207us) server_negotiation.cc:407] Connection header received
0926 11:34:25.043497 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:25.043500 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:25.043551 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:25.043602 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:25.044643 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:25.045362 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:25.047342 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:25.047722 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:25.047911 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:25.055078 (+ 7167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:25.055082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:25.055085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:25.055114 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:25.057100 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:25.059922 (+ 2822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:25.059926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:25.059927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:25.060036 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:25.060903 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:25.060907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:25.060908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:25.061113 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:25.061231 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:25.061918 (+ 687us) server_negotiation.cc:299] Negotiation successful
0926 11:34:25.062042 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 11:34:25.064446 23282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f91a563b093440fb3634fc7a721bc59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb52c8bd23a349cfa38c652bbcb1c044. Latest consensus state: current_term: 1 leader_uuid: "eb52c8bd23a349cfa38c652bbcb1c044" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb52c8bd23a349cfa38c652bbcb1c044" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } } peers { permanent_uuid: "7a0e09799e7a43299d6e85f35e540def" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43369 } } peers { permanent_uuid: "7f91a563b093440fb3634fc7a721bc59" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35251 } } }
I20260926 11:34:25.064569 23282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f91a563b093440fb3634fc7a721bc59 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:34:25.066056 23152 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7a0e09799e7a43299d6e85f35e540def [term 1 FOLLOWER]: Refusing update from remote peer eb52c8bd23a349cfa38c652bbcb1c044: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 11:34:25.066823 23276 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eb52c8bd23a349cfa38c652bbcb1c044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a0e09799e7a43299d6e85f35e540def" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 11:34:25.067617 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:25.041226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35362 (local address 127.0.98.61:43369)
0926 11:34:25.041336 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:25.041340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:25.049984 (+ 8644us) server_negotiation.cc:407] Connection header received
0926 11:34:25.054660 (+ 4676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:25.054675 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:25.054733 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:25.054787 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:25.057818 (+ 3031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:25.058451 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:25.060036 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:25.060352 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:25.060430 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:25.063604 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 11:34:25.063609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 11:34:25.063612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 11:34:25.063643 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 11:34:25.065545 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:25.066091 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:25.066095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:25.066096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:25.066187 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 11:34:25.066559 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 11:34:25.066562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 11:34:25.066564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 11:34:25.066781 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 11:34:25.066896 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:25.067321 (+ 425us) server_negotiation.cc:299] Negotiation successful
0926 11:34:25.067451 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 11:34:25.070151 23282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f91a563b093440fb3634fc7a721bc59 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eb52c8bd23a349cfa38c652bbcb1c044" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb52c8bd23a349cfa38c652bbcb1c044" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } } peers { permanent_uuid: "7a0e09799e7a43299d6e85f35e540def" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43369 } } peers { permanent_uuid: "7f91a563b093440fb3634fc7a721bc59" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35251 } } }
I20260926 11:34:25.070268 23282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f91a563b093440fb3634fc7a721bc59 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:34:25.070549 23314 mvcc.cc:205] Tried to move back new op lower bound from 7333570416875597824 to 7333570416808517632. Current Snapshot: MvccSnapshot[applied={T|T < 7333570416875597824}]
I20260926 11:34:25.071350 23263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a0e09799e7a43299d6e85f35e540def [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb52c8bd23a349cfa38c652bbcb1c044. Latest consensus state: current_term: 1 leader_uuid: "eb52c8bd23a349cfa38c652bbcb1c044" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb52c8bd23a349cfa38c652bbcb1c044" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } } peers { permanent_uuid: "7a0e09799e7a43299d6e85f35e540def" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43369 } } peers { permanent_uuid: "7f91a563b093440fb3634fc7a721bc59" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35251 } } }
I20260926 11:34:25.071456 23263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a0e09799e7a43299d6e85f35e540def [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:34:25.071668 23312 mvcc.cc:205] Tried to move back new op lower bound from 7333570416875597824 to 7333570416808517632. Current Snapshot: MvccSnapshot[applied={T|T < 7333570416875597824}]
I20260926 11:34:25.072844 23305 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb52c8bd23a349cfa38c652bbcb1c044 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eb52c8bd23a349cfa38c652bbcb1c044" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb52c8bd23a349cfa38c652bbcb1c044" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } } peers { permanent_uuid: "7a0e09799e7a43299d6e85f35e540def" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43369 } } peers { permanent_uuid: "7f91a563b093440fb3634fc7a721bc59" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35251 } } }
I20260926 11:34:25.072988 23305 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb52c8bd23a349cfa38c652bbcb1c044 [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:25.077311 23310 catalog_manager.cc:1347] Generated new cluster ID: dec72441a11649a6bd88e65ca008e694
I20260926 11:34:25.077389 23310 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 11:34:25.081696 23263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a0e09799e7a43299d6e85f35e540def [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eb52c8bd23a349cfa38c652bbcb1c044" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb52c8bd23a349cfa38c652bbcb1c044" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } } peers { permanent_uuid: "7a0e09799e7a43299d6e85f35e540def" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43369 } } peers { permanent_uuid: "7f91a563b093440fb3634fc7a721bc59" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35251 } } }
I20260926 11:34:25.081825 23263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a0e09799e7a43299d6e85f35e540def [sys.catalog]: This master's current role is: FOLLOWER
I20260926 11:34:25.081969 23322 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb52c8bd23a349cfa38c652bbcb1c044 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eb52c8bd23a349cfa38c652bbcb1c044" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb52c8bd23a349cfa38c652bbcb1c044" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } } peers { permanent_uuid: "7a0e09799e7a43299d6e85f35e540def" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43369 } } peers { permanent_uuid: "7f91a563b093440fb3634fc7a721bc59" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35251 } } }
I20260926 11:34:25.082195 23322 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb52c8bd23a349cfa38c652bbcb1c044 [sys.catalog]: This master's current role is: LEADER
I20260926 11:34:25.114846 23310 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 11:34:25.116263 23310 catalog_manager.cc:1504] Loading token signing keys...
I20260926 11:34:25.134050 23310 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb52c8bd23a349cfa38c652bbcb1c044: Generated new TSK 0
I20260926 11:34:25.135062 23310 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 11:34:25.173182 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:34:25.169738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50238 (local address 127.0.98.62:42743)
0926 11:34:25.169873 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:25.169878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:25.171749 (+ 1871us) server_negotiation.cc:407] Connection header received
0926 11:34:25.172204 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:25.172208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:25.172261 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:25.172315 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:25.173022 (+ 707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50238: BlockingRecv error: recv got EOF from 127.0.0.1:50238 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 11:34:25.174583 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:34:25.170691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35366 (local address 127.0.98.61:43369)
0926 11:34:25.170809 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:25.170814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:25.173687 (+ 2873us) server_negotiation.cc:407] Connection header received
0926 11:34:25.173860 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:25.173863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:25.173912 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:25.173963 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:25.174453 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35366: BlockingRecv error: recv got EOF from 127.0.0.1:35366 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 11:34:25.176009 23260 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:34:25.171357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37022 (local address 127.0.98.60:35251)
0926 11:34:25.171471 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:25.171475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:25.174967 (+ 3492us) server_negotiation.cc:407] Connection header received
0926 11:34:25.175234 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:25.175237 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:25.175289 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:25.175347 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:25.175888 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37022: BlockingRecv error: recv got EOF from 127.0.0.1:37022 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 11:34:25.180605 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:34:25.178445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35378 (local address 127.0.98.61:43369)
0926 11:34:25.178561 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:25.178565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:25.179590 (+ 1025us) server_negotiation.cc:407] Connection header received
0926 11:34:25.179854 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:25.179858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:25.179936 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:25.179988 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:25.180485 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35378: BlockingRecv error: recv got EOF from 127.0.0.1:35378 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 11:34:25.185782 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:34:25.184303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35382 (local address 127.0.98.61:43369)
0926 11:34:25.184410 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:25.184414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:25.184871 (+ 457us) server_negotiation.cc:407] Connection header received
0926 11:34:25.185052 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:25.185057 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:25.185111 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:25.185162 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:25.185677 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35382: BlockingRecv error: recv got EOF from 127.0.0.1:35382 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 11:34:25.191473 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:34:25.189989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35384 (local address 127.0.98.61:43369)
0926 11:34:25.190093 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:25.190098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:25.190570 (+ 472us) server_negotiation.cc:407] Connection header received
0926 11:34:25.190765 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:25.190769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:25.190821 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:25.190872 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:25.191366 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35384: BlockingRecv error: recv got EOF from 127.0.0.1:35384 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 11:34:25.199468 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:34:25.197799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35394 (local address 127.0.98.61:43369)
0926 11:34:25.197942 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:25.197946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:25.198427 (+ 481us) server_negotiation.cc:407] Connection header received
0926 11:34:25.198689 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:25.198692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:25.198761 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:25.198813 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:25.199353 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35394: BlockingRecv error: recv got EOF from 127.0.0.1:35394 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260926 11:34:25.212147 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:34:25.210306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35398 (local address 127.0.98.61:43369)
0926 11:34:25.210439 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:25.210444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:25.211030 (+ 586us) server_negotiation.cc:407] Connection header received
0926 11:34:25.211272 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:25.211276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:25.211334 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:25.211391 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:25.212016 (+ 625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35398: BlockingRecv error: recv got EOF from 127.0.0.1:35398 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 11:34:25.235133 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:34:25.230845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35414 (local address 127.0.98.61:43369)
0926 11:34:25.230976 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:25.230980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:25.233868 (+ 2888us) server_negotiation.cc:407] Connection header received
0926 11:34:25.234131 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:25.234135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:25.234199 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:25.234285 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:25.235001 (+ 716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35414: BlockingRecv error: recv got EOF from 127.0.0.1:35414 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 11:34:25.271051 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:34:25.269923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35418 (local address 127.0.98.61:43369)
0926 11:34:25.270077 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:25.270081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:25.270207 (+ 126us) server_negotiation.cc:407] Connection header received
0926 11:34:25.270329 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:25.270332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:25.270392 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:25.270494 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:25.270926 (+ 432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35418: BlockingRecv error: recv got EOF from 127.0.0.1:35418 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 11:34:25.338351 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:34:25.337268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35428 (local address 127.0.98.61:43369)
0926 11:34:25.337394 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:25.337398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:25.337568 (+ 170us) server_negotiation.cc:407] Connection header received
0926 11:34:25.337688 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:25.337691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:25.337742 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:25.337828 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:25.338242 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35428: BlockingRecv error: recv got EOF from 127.0.0.1:35428 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 11:34:25.469877 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:34:25.468683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35436 (local address 127.0.98.61:43369)
0926 11:34:25.468827 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:25.468831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:25.469043 (+ 212us) server_negotiation.cc:407] Connection header received
0926 11:34:25.469147 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:25.469149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:25.469200 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:25.469285 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:25.469730 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35436: BlockingRecv error: recv got EOF from 127.0.0.1:35436 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 11:34:25.729452 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 11:34:25.728155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35452 (local address 127.0.98.61:43369)
0926 11:34:25.728330 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:25.728335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:25.728434 (+ 99us) server_negotiation.cc:407] Connection header received
0926 11:34:25.728573 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:25.728576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:25.728662 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:25.728788 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 11:34:25.729332 (+ 544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35452: BlockingRecv error: recv got EOF from 127.0.0.1:35452 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 11:34:26.040625 23275 catalog_manager.cc:1259] Loaded cluster ID: dec72441a11649a6bd88e65ca008e694
I20260926 11:34:26.040709 23275 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7a0e09799e7a43299d6e85f35e540def: loading cluster ID for follower catalog manager: success
I20260926 11:34:26.042361 23275 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7a0e09799e7a43299d6e85f35e540def: acquiring CA information for follower catalog manager: success
I20260926 11:34:26.043500 23275 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7a0e09799e7a43299d6e85f35e540def: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 11:34:26.054366 23304 catalog_manager.cc:1259] Loaded cluster ID: dec72441a11649a6bd88e65ca008e694
I20260926 11:34:26.054464 23304 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7f91a563b093440fb3634fc7a721bc59: loading cluster ID for follower catalog manager: success
I20260926 11:34:26.056136 23304 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7f91a563b093440fb3634fc7a721bc59: acquiring CA information for follower catalog manager: success
I20260926 11:34:26.057287 23304 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7f91a563b093440fb3634fc7a721bc59: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 11:34:26.247450 23388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:26.243385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35466 (local address 127.0.98.61:43369)
0926 11:34:26.243633 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:26.243637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:26.243655 (+ 18us) server_negotiation.cc:407] Connection header received
0926 11:34:26.243820 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:26.243823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:26.243951 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:26.244054 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:34:26.244364 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:26.244920 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:26.245908 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:26.246163 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:26.246261 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:26.246544 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:34:26.246990 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:34:26.247120 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:26.247162 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 11:34:26.247215 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":78,"threads_started":1}
I20260926 11:34:26.254691 23394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:26.250578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37030 (local address 127.0.98.60:35251)
0926 11:34:26.250785 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:26.250788 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:26.250801 (+ 13us) server_negotiation.cc:407] Connection header received
0926 11:34:26.250948 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:26.250950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:26.251068 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:26.251190 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:34:26.251437 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:26.252064 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:26.252996 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:26.253252 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:26.253339 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:26.253675 (+ 336us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:34:26.254147 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:34:26.254264 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:26.254410 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 11:34:26.254493 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":78,"threads_started":1}
I20260926 11:34:26.262171 23388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:26.257919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35470 (local address 127.0.98.61:43369)
0926 11:34:26.258087 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:26.258091 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:26.258511 (+ 420us) server_negotiation.cc:407] Connection header received
0926 11:34:26.258733 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:26.258736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:26.258782 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:26.258865 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:34:26.259164 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:26.260002 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:26.261073 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:26.261347 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:26.261468 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:26.261665 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:34:26.261744 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:34:26.261821 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:26.261945 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 11:34:26.262001 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 11:34:26.262677 23394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:26.258478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37046 (local address 127.0.98.60:35251)
0926 11:34:26.258612 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:26.258616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:26.258849 (+ 233us) server_negotiation.cc:407] Connection header received
0926 11:34:26.259070 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:26.259075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:26.259152 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:26.259238 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:34:26.259467 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:26.260018 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:26.261375 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:26.261691 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:26.261788 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:26.262181 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:34:26.262258 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:34:26.262310 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:26.262494 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 11:34:26.262550 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 11:34:26.263424 23401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 11:34:26.257869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50248 (local address 127.0.98.62:42743)
0926 11:34:26.258229 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0926 11:34:26.258232 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 11:34:26.258493 (+ 261us) server_negotiation.cc:407] Connection header received
0926 11:34:26.258673 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 11:34:26.258680 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 11:34:26.258869 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 11:34:26.258997 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 11:34:26.259344 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:26.260154 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:26.261567 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 11:34:26.261931 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 11:34:26.262033 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 11:34:26.262462 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 11:34:26.262838 (+ 376us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 11:34:26.262954 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 11:34:26.263226 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 11:34:26.263291 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":63,"threads_started":1}
I20260926 11:34:26.266314 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 23035
I20260926 11:34:26.273025 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 23110
I20260926 11:34:26.278568 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskF2HClO/build/debug/bin/kudu with pid 23185
2026-09-26T11:34:26Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3093 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6165 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (285984 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST